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This handbook ou\:lines procedures for planni:ig and 
conductinq follow-up studies of clients involved in chemical 
dependency programs. It is organized into eight chapters. Chapter I 
discusses purposes and applications ot client follow-up studies. 
Studies mav be used in program evaluation , program design, and staff 
development. Also, the study itself can provide an extension of 
program services. Chapter II describes the content of the study. 
Topics include outlining the areas to be explored, organizing the 
areas into their fundamental components, and developing a questioning 
strategy that reliably probes each area. Chapter III focuses on 
sampling procedures, confidentiality, and additional sources of data 
to supoiement the self-report of clients. Chapter IV considers 
procedural and cost arranqemen*:s with reference to comprehensiveness 
of the study^ personnel requirements, confidentiality, financial 
resources, and consultation. Chapter v describes operating procedures 
such as compiling records foir administering the study, locating and 
informina clients of the study, and pretest5.ng study procedures. 
Chapter vi covers phases of data collection: planning for interviewer 
recruitment, selecting and training interviewers, and supervising 
data collection. Chapter VII discusses both computerized and 
non-computerized methods of arranging data for analysis. The final 
chapter outlines procedures for writino an evaluation report. (KC) 
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PREFACE 



This Handbook was prepared under contract to Minnesota's Department of 
Public Welfare, Chemical Dependency Programs Division. It was produced 
chapter-by-chapter over a year's time, and distributed to over 225 
chemical dependency programs throughout Minnesota as each chapter came 
off the press. 

The major purpose of the Handbook is to guide program staff in the planning 
and conduct of a client follow-up study. Such studies are sometimes very 
complex forms of social science research, but this Handbook is directed 
primarily to the program administrator, whom we assume to be a layper-.n 
in the fields of social science. It is directed as much to the manaaement 
of a follow-up project as to the issues of adequate research design. 

A secondary purpose of the Handbook is to facilitate discussion within 
the program on aspects of program effectiveness. A client follow-up - 
study is a form of program evaluation activity that provides information 
on how people benefit from participating in the program. Chapters I, II, 
and VIII are especially useful for provoking thought on how the program ' 
"impacts" on clients, and how the program might become more responsive 
to the needs and strengths of program participants. 

This Handbook was begun after two years' intensive work in the conduct 
of three major follow-up studies, each with the support of DPW. One 
concerned Minnesota's extensive system of "detox centers." Another 
concerned 11 early intervention programs directed at "underserved 
populations" in Hennepin County. And the third involved four residential 
treatment programs and halfway houses in Duluth. 

We tried in this Handbook to pull together the wisdom gained in doing 
this kind of study, hoping to advance the "state of the art" of evaluation 
in Minnesota's chemical dependency programs. We strongly believe that 
programs can improve then^-lves dramatically by paying attention to their 
actual impact on the people they serve, and by changing their approach 
accordingly. This Handbook attempts to guide that effort. 
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CHAPTER I. PURPOSES AND APPLICATIONS OF CLIENT FOLLOW-^UP STUDIES^ 

by 

Steven E. Mayer, Judie J. Steiner, & David Wnlonick 
Rainbow Researcii, Inc. 
Minneapolis, Minnesota 

A "program" could be defined as an organization designed to provide 
services to a definable group of people ("clients") so that a certain kind 
of benefit or outcome results. The purpose of program evaluation is to learn 
whether the services were provided appropriately, x^^hether the clients realized 
the intended benefits, and at what cost. Program evaluation is a process 
that illucidates the value and costs of a program's efforts to produce the 
desired outcomes, whether they be changes in neople or systems. 

Evaluation can focus on many different aspects of a progrnTiiVs endonvors: 
the management of fiscal resources, the quality of its personnel, its image 
in the community, its attempts to help people, and the way people are actually 
helped. 

Professional evaluators typically classify evaluation activities as 
"process evaluation" or "outcome evaluation," Process evaluation is primarily 
concerned with the appropriate management of resources and with the process 
of providing services. It involves such things as financial management, 
contract compliance, client flow, adherence to professional standards, and 
implementation of program design. In short, process evaluation looks at whethe 
services are being delivered competently and appropriately. 

Outcome' evaluation is concerned with discovering whait happens to clients 
^s a result of receiving services from the program. It tries to ansx^^cr the 

■^This chapter of "Evaluation Tools & Procedures"' was prepared under conttact 
#29055-29056 with the State of Minnesota, Department of Public Welfare. 
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question, did the person (or the situation) change in the intended way? 
The goal of outcome evaluation is to determine what impact the services 
had on those who received them; it tries to reconcile the outcomes actually 
experienced with the objectives of the program. 

In the chemical dependency field, the outcomes expected from parti- 
cipation in a program can often be identified from an examination of the 
program's objectives. Detoxification programs have as their objectives the 
safe withdrawal from dangerous states of intoxication, and engagement of the 
detoxified person in an appropriate diagnostic and referral procedure. An 
outcome evaluation of this program would ask whether clients were effectively 
and safely detoxified, and whether clients were diagnosed accurately and 
provided an appropriate referral. (The astute reader will notice that we 
have refrained from defining "effectively**, ''safely*', "accurately", and 
"appropriate"* These terms will be discussed in Chapter 2; the intent hero is 
merely to describe the focus of outcome evaluation.) 

Similarly, early intervention programs that target on traditionally 
underserved populations often have as their goals an accurate chemical dependency 
assessment, and a referral appropriate to the special cultural needs of their 
clients. Outcome questions would then ask, was the referral "culturally 
appropriate", was it appropriate to the seriousness of the chemical misuse, 
and did the use of services result in prevention of problems associated with 
later stages of chemical dependency? 

As a final example, residential treatment programs are often established 
to provide services that enable their clients to recover from the ravages of 

7 
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chemical dependency and lead lives of quality sobriety. An outcome 
evaluation would ascertain whether these people were now sober, and whether 
their lives have Improved qualitatively ns a result of th. treatment 
provided by the program. 

Outcome evaluation, then, insists that the proof is in the pudding. 
We have only half the picture when we know that funds wore not mismanaged, 
or that the facilities passed inspection, or that staff were properly certified, 
or that clients received 2.3 hours of counseling a day, or that 70% left the 
program with staff approval. 

The other half is concerned, very bluntly, with the question, ''So 
What?" Is the person now detoxified? Is the person no longer abusing chemicals 
and is his life now more comfortable? Did the person change his chemical 
In such a way that he is causing fewer problems for himself or other.s? 



use 



The primary method for answering these questions is the follow-up study, 
in Which former clients are located and questioned concerning their attainment 
of the program's objectives. The intent is to learn whether people exposed 
to the program did change in the intended ways, whether these changes were 
beneficial, and whether any of the changes can be attributed to their parti- 
cipation in the program. Actually, these questions are exceedingly difficult 
to answer, and the issue of research design could fill volumes. Any one 
evaluation attempt must be limited in scope and well-planned. 

The purpose of the remaining chapters in this handbook that deal with 
outcome evaluation is to elaborate on some tools and procedures that were 
recently used in a series of demonstration follow-up studies sponsored by the 
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Chemical Dependency Program Division of Minnesota's Department of Public 
Welfare, and which should be useful in the planning and administration of 
client follow-up studies. 

APPLICATIONS OF FOLLOW-UP STUDIES 

Knowing the impact of services on clients, as ascertained by a follow- 
up study, has many uses. 

Follow-Up as Contributing to Program Evaluatio n 

From a knowledge of outcomes attributable to program participation, a 
program can: 

Learn whether objectives are being met . 

Most programs have had to specify their program objectives to receive 
funds, a license, or blessings from the Board, Often, these objectives refer 
to inducing program participants to change in ways that should be noticeable 
some time after leaving the program. A frequent objective of Minnesota's 
treatment facilities is for the client to attain a chemically-free (intoxicating 
chemicals) life style following treatment. Follow-up informs the program to 
what extent this objective is achieved. 

Learn impact of services . 

Aside from the benefits specifically intended by the program, there are 
often other outcomes that occur, some beneficial and others not. 

A major problem in, the chemical dependency field is that we know very 
little about what happens to people as a result of services, or even what happens 
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to people following services. The phenomenon of active chemical misuse 
and dependency is tremendously complex, involving large portions of person '.s 
life. It seems reasonable to assume that life followinj?, service is just aw 
complex, and the changes people make are highly individual. It would he- 
very helpful for a program to know just wliat outcomes do occur fol lowin.L; 
participation in the program. 

Obviously, if people have not changed at all, tho nroKram has had 
negligible impact. If people have changed in beneficial ways, and they bolieve 
that the program was responsible for these changes then the program has iiad 
positive impact. In contrast, if people have changed in harmful, ways and 
they attribute these changes to the program, then the program has had negative 
impact . 

Compare different treatment approaches . 

The reports of people participating in one kind of treatment approach 
can be compared with those in other approaches, to learn their relative- 
strengths and weaknesses. Comparing outcomes of two approaches, though, is 
especially risky, since one has to be sure that differences in outcomes are 
actually due to differences in the treatment. Very often, different programs 
select different kinds of clients, with varying degrees of nathology, motivations, 
strengths and weaknesses. These, and a variety of other factors, could 
explain the disparity in outcomes. l,mile client follow-up lays an appropriate 
basis for comparing treatments, the techniques for this kind of field experiment 
are complex, and generally are not discussed in this manual. 
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Learn about the ^program' s strong and weak points . 

The main value of asking clients about the changes they have experienced 
and their reactions to the program is that the program receives feedback on 
its own efficacy. Feedback is an essential ingredient to learning a complex 
task, such as how to provide appropriate services. Learning without feedback 
or knowledge of results is not possible. One cannot learn to drive a car 
while blindfolded, nor can one learn arithmetic without eventually seeing the 
right answers. Similarly, one cannot learn how to provide appropriate chemical 
dependency services without seeing the results. Doing follow-up is a way to 
see the results, and not getting these results is like operating with a 



A knowledge of the program's strong and weak points allows the program 
to rectify the weak points and augment the strong noints. 

If follow-up questions are asked about different areas of life-adjustment, 
the program can see in which areas a person, or groups of people, are having 
trouble. Knowing which areas of life are especially troublesome to its clients 
enables a program to address these areas more meaningfully. It may include 
the way it does outreach to attract certain kinds of people, the way it 
screens clients into appropriate programs, how it writes treatment plans to 
deal with those strong or weak areas, and how it does aftercare planning to 
provide for those areas after the person leaves. 

For example, upon hearing that clients of a certain program have had a 
particularly difficult time making adjustments in the area of their sexuality, 
the program can make the following basic changes: 



blindfold. 



F ollow-Up as Contributing; to Program Design 
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1) Screen people with sexual, cliff icul.ties into a more 
appropriate or additional, program. 

2) Include. something in the program itself in the area of 
sexuality. 

3) Include something in the aftercare plan that addresses 
sexuality. 

These three options may not be equally desirable, or easy to do from 
a programming standpoint, nor are they likely to produce the same results. 
They do, however, illustrate the major junctures in a program whore chan,;es 
can be implemented. 

All three of these courses of action require greater sensitivity to 
problem areas when client histories are being taken, so that treatment plans 
can be more responsive, and more realistic expectations can be communicated 
between counselor and client. 

Learning that a program is especially good with some well -defined problems 
or life-situations could lead it to specialize, where it actively seeks out 
people with those particular problems/situations. The program would then be 
an especially valuable resource because it reduces the number of poor or 
inappropriate referrals. 

Clients are often poorly informed about the nature of chemical dependency 
services and have unrealistic expectations of treatment. The program's 
knowledge of its own efficacy should become the basis for describing the program 
to prospective clients. In this way programs and clients will have more realistic 
expectations of each other. 

ERIC ^2 
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Follow-iap data are valuable to staff because they provltle: 

Knowledge of results . 

It is difficult for a counselor to know how well he did with a client 
until he knows how the client is doing after treatment. Unless a counselor 
gets this kind of closure, all his actions are based on little more than 
intuition. Even if a counselor's professional training indicates that a 
particular action is warranted, he is still acting without confirmation until 
the outcome is known. 

Appreciation of individuality . 

Each client is different, even though they may come into the program 
with very similar problems. Unless these differences are kept actively in 
mind, counselors could tend to treat everyone alike, which is clearly un- 
realistic and unproductive. Follow-up data reveal to counselors the ways in 
which their actions were actually received by clients, and the benefits derived 
from them. Clients' perceptions of what they receive are usually quite 
different from counselors' perceptions of what they give. I/hile this may be 
a threatening and humbling ':xperience for counselors, their practice as 
clinicians can only benefit. 

Increased staff competence . 

Knowledge of results provides the basis for learning. It also alloxvs 
people to feel and assess their own grov'^th as therapists and administrators. 
It leads to increased competence, which is inherently satisfying and also an 
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obvlous asset to the procram. Additionally, it assists better planning; 
of one's own continuing education. 

Since doing follow-up involves personal contact with the client after 
having left the program, there is an opportunity for something positive in 
this contact alone. 

Closi tre for clients . 

A follow-up interview provides an excellent opportunity for the olicnt 
to process and summarize his own changes. In our experience, many clfrnts 
expressed great satisfaction for the opportunity to actively refloct on all that 
has changed since services, and to speculate on the role of the program in 
bringing about those changes. 

Provide referral or assistance . 

It might happen that a follow up contact provides the client with the 
opportunity to ask for additional help. From the program's perspective a 
referral or an invitation to participate in an aftercare program could be extended. 

Follow-up is not aftercare . 

Often follow--up is confused with aftercare. They are similar only in that 
they both involve contact with the client after leaving the program, and an 
assessment of how the client is doing. However, the intent is different enough 
that it would be nearly impossible to satisfy the purposes of follow-up and 
aftercare simultaneously. In aftercare, the program wants to help the client; 
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in follow-up the program wants to learn from the client. Aftercare is 
pursuasive and therapeutic; follow-up is thoughtful and reflective. 

VJhile an aftercare contact may be scary and confrontive to someone 
who has received chemical dependency services, a follow-up contact is 
inquiring, non-threatening, and not attempting tn rliange the person. In our 
studios, we were continually impressed by the honesty and lack of denial that 
was apparent in former clients' responses. We think it is because our 
purpose was to learn about the person's experiences and not to push the 
program on them. 



Knowledge of the program's effectiveness is useful information to 
communicate to others. 

C ommu n i cati on to_ program mon 1 tors . 

Presumably there are those people or groups wanting documentation of 
the effectiveness of programs. Providing knowledge of the outcomes of program 
services is useful to boards of directors, advisory boards, consumer groups, 
referral sources, lawmakers and bureaucrats. 

Communication to referral sources . 

Knowledge that a program is especially good with people with certain 
kinds of problem:! or experiences or life situations should be communicated to 
those in contact with those people/situations, so that appropriate referrals 
from that source will result. 



Fbllow-Up as Co n tributing to Public Relations 
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Commun i cation to the ge neral public o r potential clien ts . 

Thf Hi rfn;',tliH (; f n p;i rt 1 cii 1 ;ir p r())',r;uii con i cl Uc l>ro.uli';ist w I tir 1 v , so 
that potential consumers can make n more infornuul choict? of programs and 
have more realistic expectations of tlie services. 

Communication to service delivery planners . 

Knowledge of the strengths of a particular program is valuable to 
planners who are concerned that the needs of their constituencies are being 
met. It will help reduce duplication of services and assure that effective 
services be delivered inexpensively. Outcome data help to ensure tiic program's 
rlr.htfiil place Jn tho continuum of c.aro. 

WHAT CAN BE LEARNED KROM CLIENTS? 

There are many kinds of data one can collect from former clients and 
these will be the major topic of Chapter 2. This brief introduction will 
outline the most signif cant areas of feedback that can be elicited from clients. 

Clients can tell y ou about changes they have exper ienced . 

This is of course the major purpose of follow-up. For chemical dependency 
direct service programs, it is especially important to know; 

1) How the person has changed his pattern of chemical use, 
including extent of use, ''set and setting" for use, and 
consequences of use. 

2) Whether the problems that had been associated with use at 
admission are now less serious. A good history taken for 
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treatment planning will reveal the extent that ciiemlcal use 
has affected the person's family ILfe, work, social life, 
health, finances, spirituality, emotional health, schooling* 
and relations with the law. At follow-up, clients can tell 
you whether these problems have changed and what the program 
provided to facilitate this change. 

3) In what ways the person has changed his lifestyle and behavior. 
Presumably what a person learns from services is how to do 
things differently, so that Further problems nrc proventod. 
Follow-up can show us what people are doing differently, whether 
the program helped them to learn that, and whether that is 
helpful. The program would want to know whether people are 
following the program's prescription, and with what consequences. 
Obviously, the program learns about the merits of its prescription 
from seeing the consequences. 

Clients c_anj:ell you abou t thei r reactions t o the pro gram. 

They can tell you how they felt about the staff, the facilities, tiu» 
program's philosophy, the daily routine, the way they were treated, and 
experiences that were positive and that were negative. The experience of 
treatment is quite painful for many people but some feel that the experience 
was meaningful to them, while others do not. The task is to learn what was 
meaningful and why. 

Clients can tell you whether they "bought into the program . " 

This is similar to the above, but the focus is more on what they did 
subsequent to the program. A major category of questions concerns the use of 
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other services or programs following treatment. A problem arises in 
interpretation: if a person goes to treatment a second time, is tluit 
good or bad? Does it mean he is getting better, or worse, cr beginning to 
become dependent on services? 

Clients can offer recommendations . 

With the benefit of hindsight and their own experiences former clients 
are often quite willing to recommend certain changes in the program or its 
personnel . 

OVERVIEW OF A FOLLOW-UP STUDY 

There are many forms a follow-up study can take and many features that 
require planning. The next several chapters of this Handbook will try to 
answer the following questions: 

1) Who should be f ollowed--up? Former clients themselves are an 

excellent source of data. But each and every one? Can a sample 

be taken? Should those who left the program before the program 

said they were finished be included? Should "significant others" 

be asked to corroborate the story? How else can stories be corroborated? 

2) I'Jhat should they be asked? The study objectives should be 
articulated first, and then translated into questions that will 
satisfy those objectives. A well phrased question will elicit reliable 
answers and invite the respondent to be candid and honest. 

3) Iflien should they be asked? Should people be contacted three months 
after leaving the program? Six months? Two years? Two weeks? More 
than once? Is there a bad time? 

Z 8 
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How should they be asked? Outcomes can be assessed in face- 
to-face interviews, telephone interviews and mailed questionnaires. 
There are advantages and disadvantages to each. The following 
chapters deal entirely with interview follow-up. 

Who should do the asking? Many questions of "'scientific integrity'' 
and "professional integrity" arise in followiip studies. Tf the 
program is in charge of its own evaluation, isn*t it like asking the 
fox to guard the chicken coop? But if an outside agent does the 
interviewing, isn't this a violation of trust developed between 
counselor and client? A chapter on contractual models of follow-up 
will address these questions. 

What are the qualities of a good interviewer? How does the inter- 
viewer's own attitude about chemical use affect the interview? What 
training does an interviewer need? 

Hov. should clients be contacted? l^^^at rules of client confidentiality 
and data privacy apply? How should interviews be scheduled? What 
can be done to keep track of the sample? 

How should the data be tabulated? Are computers really necessary? 
Are fancy statistics needed, or is simple counting all that is 
necessary (once you have figured out what to count)? 

I^at can be done with all those tables? No one likes to read data 
tables, but how can they be avoided? \Jho should help interpret the 
data? How to conduct staff meetings around the results of the follow-up. 
How to report data to satisfy accountability and public relations purposes. 

19 
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A DISCLAIiMER 

Our mandate, in writing this Handbook, Is to slinre wluit we know nlunit 
doing follow-up to assess client outcomes. This is not to say that we know 
everything, only that we have learned a great deal from our experience in 
administration of follow-up studies. 

Our most recent projects involve data collection at 21 different 
programs. These were 6 detox centers, 11 early intervention programs and 4 
residential programs. Ue deliberately made the administration quite different 
in these projects (though the study purposes and the philosophy of questioning 
were similar), so that we could test out different procedures. 

As one might expect, each had good and bad points. And, of course, 
there are many procedures we did not try at all. So far, we have not dis- 
covered the "one best way" and in all likelihood, there isn't one. 

Our purpose will be to present what we did with an assessment of how 
well it worked. We will not elaborate on procedures we did not try, other 
than perhaps to introduce them and indicate why we proceeded the way we did. 



Chapter I prepared September, 1978. 
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CHAPTER II. PLANNING A FOLLOW UP STUDY: CONTENT^ 

by 

Steven E. Mayer, Ph.D., David S. Walonick & Judie J. Steiner 

Rainbow Research, Inc. 
Minneapolis, Minnesota 

A client follow-up study is intended to give information useful for 
program evaluation, program design, staff development, and public relations, 
as discussed in- Chapter 1. The major objective of the follow-up study is 
to discover to what extent clients experienced the intended effects of the 
program. 

In fulfilling this objective, the follow-up study designer should... 

1) ...outline the areas to be explored in the follow-up study; 

2) ...analyze each of these areas into their fundamental components; 

3) ...develop a questioning strategy that reliably probes each 

of these areas. 

This chapter explores each of these steps. The first section discusses 
considerations in constructing a list of the intended effects of the program. 
The second section of the chapter describes a procedure for defining "Effectiveness" 
and other big words. The third section discusses some of the issues in formu- 
lating good questions for assessing client outcome areas. The fourth section 
gives examples of typical evaluation questions that chemical dependency programs 
might consider when developing their follow-up study objectives. Finally, a 
sample questioning strategy is presented to illustrate some of the principles 
discussed in the chapter. 
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DEVELOPING A LIST OF INTENDED PROGRAM EFFECTS 

The first step in the design of a follow-up study is to develop a 
comprehensive list of the intended effects of the program. There are several 
sources for discovering what effects a program intends for its clients. 

Stated program objectives or philosophy 

Most stated program objectives are directly or indirectly related to 
client outcomes. Examples are: "provide appropriate referrals to the continuum 
of care", "encourage abstinence from intoxicants", "provide chemically-free 
living skills", "decrease the harmful consequences of chemical -use", and 
"rehabilitate the total person". 

License requirements 

The rules for licensing chemical dependency programs in Minnesota often 
include provisions for program evaluation. At this time, only outpatient 
treatment programs are required to do follow-up (Rule 43). A section of this 
rule specifies that questions be asked to determine the clients' "chemical use 
patterns, severity of problems associated with use at intake, and lifestyle 
indicative of recovery and prevention." 

Individualized client plan 

Treatment plans, discharge plans, and aftercare plans (such as those 
suggested in program accreditation standards) should include expectations for 
certain outcomes to be experienced after services are completed. These often 
contain goals for clients, stating what behavior, activities or events would 
be considered signs of "success" for a given client. 
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Client case files 

When a client approaches a program for services, a contract between 
them is at least implicit. Client case files can be examined to discover the 
program's expectations of the client, the ways a client desires to change, 
and the expectations that the client has of the program. 

Source of referral 

Referral sources also have expectations of programs for their clients. 
A physician may refer a person to a program anticipating improvements in various 
bodily or mental functioning. A court may refer a person to a program expecting 
changes in that person's behavior with regard to the law. School systems, 
families, and employers may each hope for different kinds of outcomes. Exploring 
these various referral sources for their expectations is a good way to determine 
the intended effects of a program. 

Funding contracts and funders 

If a program is not entirely supported by clients, it usually receives 
funds from a governmental unit or foundation. Funding contracts typically specify 
that the program perform certain services and facilitate certain outcomes. They 
are often more explicit about services than about outcomes, but discussions 
with funders may give a clearer picture of the expected outcomes. 

Discussion among staff 

Often, the intended effects of a program are assumed by the staff but are 
rarely discussed in-depth within the program. It will be helpful for the staff, 
as a group, to discuss the following topics: What does your program attempt to 
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do for its clients? How do you recognize the signs of "successful" or 
"unsuccessful" outcomes in your clients? What is the difference between your 
program and another program, in terms of the benefits or outcomes it expects? 

These discussions are useful as a form of in-service training, even if 
no follow-up study is contemplated. They will clarify the program's purposes 
by helping to articulate the outcomes intended for clients, and generally 
inspire a better understanding of the program among its staff. 

A METHOD FOR, DEFINING "EFFECTIVENESS" AND OTHER BIG WORDS 

From all the previously mentioned sources, a list of intended outcomes 
can be developed. The list will probably contain a variety of terms, from very 
specific to very abstract. The abstract terms, such as "quality sobriety," 
"appropriate referral," or "lifestyle indicative of recovery" will most likely 
mean different things to different people. The next step in follow-up design 
is to develop precise definitions for these terms that will be acceptable to 
those interested in the follow-up study results. 

Discovering the meaning of abstract terms 

The problem is similar to that surrounding terms such as "truth," "freedom," 
and "democracy." We each believe we know what they mean, and yet our discussions 
with most everyone reveal that we differ on the central meaning of each term, 
on what should be included in its definition, and what should be taken as 
evidence of its presence. 

An important task for designing the content and strategy of a client 
follow-up study is the rendering of these abstract terms into more definitive 
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terms that carry greater "shared meaning." As the meaning becomes more 
specific, questions can be constructed that are direct enough to yield 
trustworthy and useful answers (in other words, increase the reliability and 
validity of the study). 

Imagine asking a client the question, "Did you get an appropriate referral 
An answer of "Yes" or "No" would not be helpful feedback to the program unless 
we were sure the client understood what we meant by the question. Obviously, 
the question as asked is not sufficient to give trustworthy and useful data; 
the notion of "appropriateness" has to be broken down into more meaningful 
pieces. 

The process of giving greater definition to abstract terms requires that 
you (the study designer) keep asking the question "What do you mean?", until... 

... the meaning becomes clear; when it is boiled down to its list 

of basic ingredients. 
... the meaning is complete; when the abstract term has been compared 

to its precise parts to see that the whole is well -represented by 

these parts. 

... the meaning is shared; when the audience of the study would understand 
the analysis of the term. 

For example, when looking at the meaning of "an appropriate referral" we 
find several components that need to be considered. In a larger sense, appro- 
priateness refers to the compatibility of a client with the referral in terms of 
ease of access to the program, and the ability of the program to relate to 
the person's situation. Breaking it down further, ease of access relates to 
cost, convenience, privacy, and personal support for access. Ability to relate 
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to the client's situation includes aspects of a cultural compatibility, 
degree of chemical involvement, personal strengths and resources of the client, 
and constraints that affect the client's commitment to the program. When 
the goal of a program is to provide appropriate referrals to its clients, 
questions should be asked to learn whether a referral was provided and in what 
ways it was appropriate. 

A comprehensive follow-up study contains questions that probe important 
aspects of each of the outcome areas intended for clients. In this way, the 
resulting data will be grounded in the meanings held by the people that will 
be affected by the study. 

m 

Setting limits to the follow-u p 

You may discover from an analysis of the intended effects that the program 
is ambitious in its goals for clients and that the outcome areas are quite 
complex. A substantial number of questions may have to be asked of clients to 
learn whether they indeed had experienced these effects. 

Generally, the more complex the area, the more time is needed in the follow-up 
contact to make a comprehensive assessment of the area. Time taken to explore 
one area means time lost to explore another. This phenomenon forces the study 
designer to decide which outcome areas need the most thorough (detailed) 
examination. The direction for the study then becomes one of exploring each of 
the outcome areas to the extent prioritized. 

Learning about effectiveness vs. making claims of effectiveness 

Programs often conduct follow-up studies to determine their "effectiveness." 
This is accomplished partly by assessing whether program clients experience the 

ERIC ^ 



II-7 



outcomes intended for them. Obviously, if a program finds that all of its 
clients experienced the intended outcomes, it could conclude the program is 
"effective" in one sense of the word. 

In another sense, a program could claim effectiveness for only those 
clients experiencing positive outcomes (as in "the program was 43% effective"), 
or for only those who follow the program's treatment plan (as in "for those 
staying with the program, the program was 76% effective"). 

And in still another sense, effectiveness cannot be claimed until it is 
shown that the program produces or facilitates positive outcomes better than, 
or with fewer negative side effects than n_o program, or an alternative program. 

It is this last meaning of effectiveness that comes closest to the scientific 
meaning of "the effectiveness of a treatment." If you study only those people 
who experienced treatment, and not those who didn't experience treatment (but 
who could have), the data will not estimate "the effect" of treatment. Procedures 
for choosing appropriate control groups will be discussed in Chapter III. 
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QUESTION AND ANSWER STRATEGIES 



The basic method of a follow-up study involves the use of a question 
and answer strategy. 2 That is, the primary data in a follow-up study consist 
of answers to questions. Considerable attention should be given to the design 
of an effective questioning strategy. The quality of the data will never be 
better than the quality of the design for asking questions and recording answers. 



A good question and answer strategy is one that... 

1) ...yields answers that are reliable . Reliable answers are 

those that are free from undesirable or irrelevant influences. 
Reliable answers are those that would be given consistently 
when asked under a variety of circumstances. Unreliable 
answers-result from peculiarities of the interview situation 
or the person's momentary feelings, or confusion over the 
meaning of the question, or inability of the answer format 
to accommodate the person's true response. Some questioning 
styles are more likely to be reliable than others. 

2) ...yields answers that are valid . Answers are valid if they 

actually and realistically assess the outcome area in question. 
To be valid, answers must first of all be reliable. The procedure 
discussed in the last section of this chapter is meant to 
enhance the validity of the questions. Validity can be shown 
when the answers are in agreement with other kinds of known 
facts that could be gathered to answer the question. For 
example, the reason a "significant other" is sometimes asked to 
corroborate (validate) another's story is because the validity 
of self-report data is sometimes in doubt. While validity is 
very difficult to test, it is known that some questioning styles 
are more likely to yield valid answers, than others. 

3) ...uses interview time efficiently . In an effort to get reliable 

and valid data, a great deal of interview time can be consumed. 
Interview time should be invested where it will yield the most 
useful resul ts. 



^There are other ways of taking samples of a person's post-program experience 
than by asking questions of that person. One can ask questions of other 
persons, make direct observations of the person without asking questions, take 
blood, urine, or body tissue samples, examine institutional records, or observe 
a person in a specially constructed role-play situation. While these techniques 
can provide valuable information to programs, they will not be discussed in 
this handbook. 
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Question formats 

There are many ways of asking questions, and recording answers. Developing 
question formats entails at least three basic considerations: 

1) Two-way vs. one-way communication. A personal interview (either 
face-to-face or on the telephone) is an example of two way 
communication. A question is asked, an answer is given, and 
clarification may be sought be either party. A written questionnaire 
is an example of one-way communication, where a question is asked and 
an answer is given, but no clarification is possible. A written 
question must clearly communicate the intended idea, since there is 
no opportunity to learn whether the communication was received accurately. 

2) The interview can be "standardized" or "clinical". A standardized 
interview asks the same questions of all respondents; the interviewer 
conducts the same interview with each client. A clinical interview 
is one that is different from one client to the next, and has the 
advantage of tailoring the interview to each client, and the disadvantage 
that answers cannot be compared from one person to the next (since the 
questions were different). A standardized interview has the advantages 
of comparability across respondents, and greater ease of analysis. 
It has the disadvantage of appearing inflexible, since it imposes the 
same structure on each respondent's story, A worthwhile but difficult 
goal is to construct a standard format that allows one to capture as 
much individuality as possible, and at the same time, permit comparisons 
from one person to the next. 
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3) Questions can be "open-ended" or "closed-ended". With open-ended 
questions^ the client's own words are recorded; with closed-ended 
questions, the client chooses the best answer from a prepared list of 
possible answers. The advantages of open-ended questions are that one 
hears rare and unique answers that may have escaped a predetermined 
list, and one keeps the special "flavor" of a personal report. They 
have the disadvantage of requiring considerable time afterwards to 
analyze and summarize the responses. Closed-ended questions have the 
advantage of rapid tabulation, and the disadvantages of appearing 
impersonal, and of requiring beforehand knowledge of the range of possible 
answers . 

Answer formats 

Along with the variety of questioning strategies, there are a number of 
ways to record answers. They fall into three major classes: categories, 
psychological units of measurement, and physical units of measurement. 

As an example of categorical assessment of a person's alcohol involvement, 
there could be the following: 

"When do you drink alcohol?" 

1 = drink whenever I feel angry or depressed 

2 = drink only on weekends 

3 = drink whenever I can 

4 = never drink alcohol 

In a category system, the numbers are used only to identify the coding 
categories; they do not signify "quantity." The difficulty with categories is 
that one must be assured that they exhaust all the possibilities, and that they 
be non-overlapping (the above categories serve as bad examples). 
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Many answer formats using psychological measurement units have been 
developed for questionnaire studies. As examples: 

a) "How often do you drink alcohol?" 

1 = never 

2 = seldom 

3 = frequently 

b) "On a 5-point scale, rate your alcohol consumption" 

1 2 3 4 5 
None Heavy 



c) Indicate your agreement to the following: "I am a heavy drinker" 

1 = Strongly disagree 

2 = Disagree 

3 = Neutral 

4 = Agree 

5 = Strongly agree 

In tlier,e examples the numbers are meant to suggest an underlying dimension 
of "magnitude of alcohol involvement"; higher numbers suggest greater alcohol 
involvement. For these kinds of ratings or rankings to be reliable, we must be 
assured that the continuum portrayed by the answer scale is realistic, and 
likely to be understood by respondents. 

Sometimes a scale can be used that expresses magnitude or quantity in 
physical units of measurement. For example, alcohol consumption can be measured 
in terms of "number of ounces of 86-proof alcohol consumed per standard unit of 
time", which uses the physical scale of weight, expressed in ounces. To use this 
scale, beer and liquor and wine concentrations of alcohol per volume must first 
be transformed into ounces of 86-proof alcohol; also respondents must be able 
to estimate accurately their use for the unit of time used in the measure. Physical 
units of quantity have not been developed for most chemicals other than alcohol. 
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Most outcome variables of interest (such as "closeness" of family, 
"harmfulness" of use, "quality" of sobriety) do not lend themselves to 
answer formats with physical units of measurement. Even reliable psychological 
measuring units are very difficult to design. The approach suggested in the 
last section of this chapter is intended to enhance the reliability of 
measurement by forcing increasingly greater clarity in the meaning of terms used 
in the question and answer formats. 

Many answer formats have been extensively researched for their ability 
to record answers reliably. Advice on these formats is available in the 
excellent publication by the Measurement Service Center at the University of 
Minnesota, "Questionnaire Brochure Series: 1. Writing Good Questions and 
Response Alternatives". Also, recommended is 3ue_s_yojijialresjJ)eji^ 
by Berdie and Anderson (Scarecrow Press, 1974). 

Already many combinations have become apparent from the above considerations. 
The choice of questioning and answering formats is a difficult one, and is 
probably guided by cost. Important costs to consider are: 

1) the cost of conducting a one-way communication process vs. a two-way 
communication process of collecting data. The cost of an interviewer 
can be high, but so can the cost of developing a questionnaire of 
acceptable reliability. 

2) the cost of analyzing the data. Closed-ended questions require time 
in the early design stages; open-ended questions require it after the 
data have been collected. If little is known about the variety of 
outcomes experienced by the program's clients, it is better to cast a 
wide net by asking open-ended questions. 

32 
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3) the cost of increased detail. As a rule of thumb, the more 

detailed an assessment that is needed, the more questions will have 
to be asked; asking several "small" questions will yield greater 
detail (and reliability) than asking one big one. The opportunity for 
detail is limited by the time allowed for asking and answering questions. 

4) the co-t of overestimating or underestimating your program's effects 
because of unreliable data. Unreliable data result frorr questions that 
produce answers that are contaminated by irrelevant influences, and 
from answer formats that do not accommodate the true variability of 
program clients. Removing these sources of unreliability is also costly, 
in terms of efforts expended during the preliminary design. The 
reliability of data cannot be improved after they have been collected. 
Conclusions based on unreliable data can also have costly repercussions 
for the program, its future clients, and society. 

Assessing the qualit y of question and answer formats 

Developing good questions is a difficult task, and even the experienced 
researcher will spend many hours contemplating the best formats and phrasings. 
If a question is even slightly ambiguous or susceptible to misinterpretation, 
it should either be redesigned or excluded entirely. Respondents' answers to 
a question will never be more clear than the question itself. There are 
several characteristics of good questions, each of which serves to increase 
reliability and validity. 

A good question is one that: 

1) ...evokes the truth. A person is more likely to tell the truth if 

the question is non-threatening. An example of a threatening question 
would be, "You didn't get drunk last week, did you?" 
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..asks for an answer on only one dimension or yardstick. For 
example, the question "How long did you go to counseling?" elicits 
the answer "3 months" or "3 times". While both of these are 
acceptable answers from one standpoint, they preclude any summarizing 
across clients. The question unintentionally elicits responses 
on two different metrics (incidence and time). 

.evokes an answer that most truly represents the person. Some 
questions are particularly sensitive to fluctuations in the person's 
mood (e.g., "How are you feeling these days?"). The answer you get 
today may be very different from the one you get in two weeks. .Other 
questions inadvertently lead the respondent to a particular answer. 
Some questions may not have appropriate answer categories. Try your 
questions out on yourself and other people under varying circumstances, 
and note the variety of responses to a question. Choose a question 
and answer format that minimizes this kind of irrelevant variability. 

.elicits analyzable answers. Even open-ended questions should strive 
to be as specific as possible, so that the job of constructing 
categories is easier. An open-ended question is not the same as a 
general question* 

.can faithfully accommodate everyone on the answer scale. If a 
category answer format is used, categories must be able to include 
all the possible responses to the question, and should not overlap. 
Including an "other" category is usually advisable. Answer formats 
using psychological or physical units of measurement should have 
enough room at the "low" end and the "high" end to accommodate 
everyone's true response. 
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6) ...produces variability or variety in responses. If everyone 

gave the same response to a particular question, little information 
is gained. Variability is needed if one is looking for differences 
between various subgroups of the sample of people interviewed. 
Without variability, differences cannot be discovered. Answer 
formats should reflect the true variability in peoples' responses 
(without introducing variability from irrelevant factors.) 

7) ...follows comfortably and logically from the questions before it. 

One should pay attention to the flow of questions, from the beginning 
to the end. It is often helpful to begin with questions that orient 
the client to the right time frame, recalling his/her admission 
to the program and the circumstances surrounding it, and to end 
by asking questions that encourage the client to summarize or get 
closure on the interview experience. 

Many of the factors that make for unreliable data can be anticipated in 
advance and be designed out of the questionnaire or interview. Others can be 
discovered only by conducting a pretest or trial run of the entire procedure, 
practicing on your client population as much as possible. It is vitally 
important that such a trial run be conducted to discover sources of unreliability; 
many of these will not be discovered except under field conditions. 

Quality control in questionnaire design 

As the question and answer strategy evolves, there should be a continual 
review of the procedure. The following list of questions will help the researcher 
assess the quality of the follow-up interview or questionnaire. 
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1) Win the answers to this set of questions assess the outcome 
area as intended? 

2) Can the answers to this question be summarized across people? Can 
Joe*s answer be compared to Mary's answer? If there is no comparability, 
there will be the "apples and oranges" problem. 

3) Can the responses be put into a data table that coirimunicates what you 
want it to? 

4) Is the time for exploring one outcome area justified by the priorities 
assigned to that particular outcome area? Is there a shorter pro- 
cedure that would yield as good results? 

5) Will this question, or set of questions, invite the person to be 
straightforward and honest? 

Ass essmen t of cha nge 

Implied in assessing client outcomes is the assessment of change in the 
person from his/her before-program experience to post-program experience. 
We are partly interested in the client's present chemical use, but we are 
more interested in knowing how this is different than before treatment. Similarly, 
we are partly interested in knowing about the client *s present status of 
problems in his/her family life, but we are more interested in learning how 
this compares to ramily life problems prior to treatment. 

Assessment of change is a difficult and technical problem. There are 
two basic approaches: 

One is to collect data at two points in time; the difference between those 
two points estimates the amount of change. Those two points could be at 
intake and follow-up, discharge and follow-up, or two different times of 

follow-up. ry 
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The second approach is to collect data at only the second point in 
time. This approach has two variations. In one, the person is asked to 
estimate the degree of change since time-one. (e.g., "How much better is 
this problem than before treatment?") In the other, the person is asked to 
estimate the incidence of a particular event both at time-one and at time-two, 
and then the researcher subtracts to determine the amount of change, (e.g., 
"How many DWI's did you get the year before treatment, and the year after 
treatment?"). 

The main advantage of collecting data at only one point in time is that 
data collection costs can be greatly reduced. The disadvantage is that 
respondents may be unable to recall accurately how things were at a previous 
point in time. Both methods are legitimate research techniques; however, cne 
should be aware that they may yield different results. It is also fundamentally 
important to consider the issue of comparison groups (see Chapter III). 

EXAMPLES OF RESEARCH OBJECTIVES 

In our research with chemical dependency programs, we have been able 
to identify several questions for which most programs would like to have answers. 
These became our study objectives in the demonstration projects recently 
conducted. They are presented here to illustrate the kinds of information 
which can be learned in a follow-up study. No attempt has been made to prioritize 
them, since each program will want to do this on an individual basis. 

1) What brings clients into programs? What are the circumstances 
surrounding admission? What are the access points, coercive 
circumstances around admission, and attitudes toward admission? 
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2) What are the characteristics of clients in the program, in 

terms of demographic data, strengths and resources, problem areas, 
alcohol and drug use patterns, and previous services related to 
the present admission? 

3) What do clients experience, and how do they assess the program 

and its procedures for admission, monitoring, assessment of problems, 
referral, treatment, and discharge? What can we learn about these 
processes from clients' reports? 

4) In what ways have clients' alcohol and drug use patterns changed? 
What are the changes in frequency, type and amount of use, in type 
of drugs used, in symptomatic use and in consequences of use? 

5) In what ways have there been changes in those life-areas that 

had been adversely effected by use prior to admission? Which changes 
do people regard as important in their own lives? 

6) What do clients do differently now? How have they changed the 
conduct of their lives? Are they doing what the program taught them, 
or what the program wanted them to do? Did clients follow through 
with referrals? Why not? What sort of referral could have been 
better? Do clients use additional services after treatment? 

7) How do different subgroups of the program's clientele compare in 

terms of outcomes and perceptions of the program? Are there differences 
between those who leave without approval and those who leave with 
approval? How do previous service histories and varied backgr*ounds 
affect peoples perceptions of the program? 

8) Do clients feel that any changes can be attributed to the program? 
How helpful was their experience with the program? 

38 
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AN EXAMPLE 

An outcome area that most chemical dependency programs are interested 
in measuring is "pattern of chemical use." In designing the instruments for 
the demonstration project with residential treatment programs, this area 
was broken down into five areas, which together assess "pattern of chemical 
use." 

These five are: 

1 ) Consumption 

2) Change in consumption since before treatment 

3) Time since last use (or length of sobriety) 

4) Presence (or absence) of signs of pathological use 

5) Change in perceived harmful ness of use since before treatment 

The following sections describe how each of these areas were assessed, 
along with a number of observations concerning the design features of each 
of the measures. 

Consumption 

The following questions could be used to measure the extent of present 
levels of chemical consumption: 

a) In the last month, how often have you drunk alcohol (beer, wine, liquor)? 
1 = not at all 2 = less than once a week 3 = 1-4 days/week 

4 = 5-7 days/week 

b) In the last month, how often have you used marijuana or hashish? 
1 = not at all 2 = less than once a week 3 = 1-4 days/week 

4 = 5-7 days/week 
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c) In the last month, how often have you used tranquilizers, pain 
killers, or barbiturates? 

1 = not at all 2 = less than once a week 3=1-4 days/week 
4 = 5-7 days/week 

d) Etc., using other chemicals or groups of chemicals. 

Using these measures, one could calculate use (or non-use) from various 
substances, in the last month. Using "the last month" as the reference point 
has the advantage of being a standard unit of time that can be measured for 
everyone, and is not confounded by the length of time people have been out of 
the program. That is, use rates could be calculated for different subsamples 
of people, such as those who have been out for one year versus those who 
have been out for two years. 

The choice of which chemicals to include, or to group together, should 
be made with the programs clientele in mind. Some programs may wish to assess 
the different opiate derivatives separately; others could combine them. Programs 
for youth may want to emphasize different chemicals (inhalents ,PCP) than 
programs for elderly (sedatives, pain killers). 

This procedure assumes that most of what is called "consumption" can be 
assessed simply by "number of occasions of use in a week", asked for different 
chemicals. Number of occasions of use in a week is probably estimated by 
people more reliably than amount of alcohol consumed; in addition, amount 
consumed is difficult to interpret, since effective dosages vary so much 
between people. 
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C^3Pje__in,j:o_nsupj)ti^ treatriiept 

It is c. 3 thing to know present chemical consumption levels. It is 
another to know how this has changed since before treatment. Questions 
paralleling present use could be asked, as follows: 

a) Is this (use of alcohol) in the last month less, the same, or 
more than before treatment? 

1 = less 2 = same 3 = more 

b) Is this (use of marijuana or hashish) less, the same, or more 
than before treatment? 

1 = less 2 = same 3 = more 

With these questions, one can learn change in consumption for each chemical 
or group of chemicals. These questions also illustrate the use of a dimensional 
continuum for recording answers, with three gradations; more than three might 
be difficult for clients to estimate reliably. 

Time since last use 

Programs often want to know the length of a person's sobriety. This 
can be measured by asking for the time since last use of various substances: 

a) How long ago was the last time you had a drink of alcohol? (use code ) 

b) How long ago was the last time you used marijuana or hashish? (use code) 

c) Etcetera, for other chemicals or groups of chemicals 

Code: 0 = never used 

1 = less than 1 week ago 
2=1-4 weeks ago 

3 = 1-3 months ago 

4 = 3-6 months ago 

5 = 6-12 months ago 

6 = 1-2 years ago 

7 = 2-3 years ago 

8 = 3-4 years ago '^l 

9 = 4+ years ago 
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An abstinent rate could be calculated with these questions. The 
program could decide which chemicals belong on the list of chemicals that 
program clients must abstain from, and decide how long a time period is 
necessary before it would confer the status of ''abstinent" on someone; it would 
then count the number of people surpassing that standard- For example, if 
the program decides that non-use of alcohol, tranquilizers, and caffeine for 
two years is what defines "abstinent", one has merely to count the number of 
respondents (assuming they fairly represent program clients) whose last use of 
alcohol, tranquilizers, and nicotine was more than two years ago. To analyze 
this question meaningfully, rates should be calculated separately for those people 
out of the program for one year, for two years, etc. 

Presence (or absence) of si gns oi^jgatholgaicj^^^^ 

Questions appearing on various "diagnostic checklists" of chemical 
dependency are used to assess the level or pathological use in those people 
who are using intoxicating chemicals at the time of admission. Presumably, 
if these signs are no longer present, recovery from chemical dependency is 
indicated. The validity of self-report on pathological signs may be questionable, 
so these answers should be taken as the "lower limit" of the probable incidence. 

Examples are: 

Sometimes, when you start drinking (or using), is it difficult to stop before 
■ becoming completely drunk (or loaded)? 1 = yes 2 = no 

Sometimes do you keep on drinking (or using) after you have promised 
yourself not to? 1 = yes 2 = no 

Do you ever wake up unable to remember what you've done the night 
before? 1 = yes 2 = no 
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Do you ever have a drink (or take a pill, or smoke dope) first 
thing in the morning? 1 ^ yes 2 = no 

Are you worried or concerned about your drinking or use of other 
drugs? 1 = yes 2 = no 



These examples also illustrate the difficulty of adapting questions 
designed for alcohol consumption to other drugs. These questions were taken 
from a previous study, so that they could be used to compare response rates. 



The questions were developed using a pathological population, with Jellinek's 

methods, and are somewhat questionable as predictors of chemical dependency. A 

question which does not assume that one person's pattern of pathology is the 

same as another's, asks: 

"We would like to know if you have gotten drunk or loaded in the 
last month. We're not asking is you were "feeling good", we're 
asking if you got drunk or loaded more tha n you probably should ? 
How many times in the last month?" 



Change in perceived responsibility of use 



For those people now using chemicals, it would be helpful to know their 
own perception of whether use is now more or le^^^*" responsible than before 
treatment. The following questions might be useful: 



a) "If you're now using, would you say that your use has become more 
or less responsible since che time you went to the program? By 
more responsible we mean has it become less dangerous or bothersome 
to yourself or to others; less responsible means it ^^as become 
more dangerous or bothersome to yourself or to othe ^* 

1 - more responsible (less dangerous or bothersome) 

2 = less responsible (more dangerous or bothersome) 

3 = Same 
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b) "If you say it is more responsible, please tell us all the 

changes you've made to make your use less dangerous to yourself 
or others. Please be specific." 

The first question has the obvious disadvantage of requiring the 
client to summarize for all chemical use. The question could be asked for each 
of several chemicals or groups of chemicals. 

The second question should facilitate a list of ways in which people 
change their use to make it less dangerous. The question was left open-ended, 
since a predetermined list would contradict the idea that changes in use in 
the direction of greater health is defined in each individual's case. 

In summary, the above measures allow an assessment of "pattern of chemical 
use" according to several different aspects of that outcome area. 
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CHAPTER III 

STUDY DESIGN: SAMPLING PROCEDURES, CONFIDENTIALITY, & OTHER DATA SOURCES 

Steven E. Mayer, David S. Walonick, and Judie J. Steiner 
Rainbow Research, Inc. 
Minneapolis , Minnesota 

Chapters I and II of this Handbook explored the uses of client 
follow-up studies, and the ways that the content of a follow-up interview 
or questionnaire could be developed to satisfy the major purpose of assessing 
client outcomes. 

The discussion of this chapter focuses on how the study designer 
can . . . 

1) specify the kinds of clients for which the program would 
like to make outcome statements; 

2) take appropriate samples of clients for inclusion in the 
follow-up study ; 

3) abide by the data privacy laws in the program's effort.s to 
obtain follow-up information from clients; 

4) consider additional sources of data to supplement the self- 
report of clients. 

The first section of the chapter discusses populations and sub- 
populations of clients, and introduces the distinction between a population 
and a sample. The second section details a process for sampling clients 
from the subpopulations within the program. The third section discusses 
the use of comparison groups to give more meaning to the results of the study. 

The fourth section discusses the data privacy laws, and presents 
the highlights of both the Federal and State regulations, along with 

This publication was prepared under contract #29055-29056 with the State of 
Minnesota, Department of Public Welfare. Chapter completed March 1979. 
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recommendations for how they might be used to guide follow-up efforts. 

The final section presents advantages and disadvantages of 
supplementing the self-report data with reports by significant others, 
or with data from official records. 

POPULATIONS AND SUBPOPULAT EONS OF CLIENTS 

Who should be included in the follow-up study sample is a major 
question. The answer depends on which people the proj>r*^ni would like to 
make outcome statements about, on the dej',reo of certainty needed to discuss 
"effectiveness", and on whether it is feasible to include them in the study. 

Populations and Subpopulations 

In ordinary language one speaks of "kinds" of people. In the medical 
and social sciences one speaks of populations and subpopulations. A popu- 
lation is a collection of people, all identifiable by a common label. Any 
particular population is defined by naming its uniqi e qualities (Guilford, 
1966). Typ ically, the population a follow-up study would moat like to learn 
about is the "population of people served by a particular program." 

The population of people served by a program can be broken down into 
any number of subpopulations. There is an almost unlimited list of sub- 
populations that could be identified within the program's total population. 
One is limited only by imagination and the availability of information 
about clients. 

Most program records allow several kinds of subpopulations to be 
identified: 

i7 
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1) clients can be subgrouped by demographic data, such as 
«igo, sex, or race. 

2) .subpopula tions may be identified by personal resources or 
life situations: there are these people with intact family 

and job, people who are unemployed, people in crisis situations, 
people in good health, etc. 

3) subpopulations may be identified by chemical use patterns: 
clients could be grouped by drug of choice, by types of problems 
experienced, or by amount of time since abusive use began. 

4) subpopulations may be identified by program service history: 
there are those people who have been in treatment before, who 
were referred by the courts, or who exi;erienced treatment but 
not aftercare. 

5) subpopulations may be identified by program administration 
variables: these might include the subpopulation of clients 
served from 1972 to 1977, elderly people served in the last year, 
or people admitted in winter rather than summer. 

Of course, subpopulations can be identified using combinations of 
the above, such as "young, unemployed males referred by the court" or '"middle- 
aged women in crisis abusing tranquilizers." Also, any one person can be a 
member of several different subpopulations, depending on how they are defined. 

Again, identification of these groups for follow-up p irposes r-^quires 
either a client record system that describes clients in thf ways needed, or 
a follow-up contact that deliberately asks questions that allow the program 
to identify subpopulations. 

'is 
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Populations and Samples 

In the procedures of research, one rarely has the entire population 
or subpopulation available for study. Since populations are often very 
large, or are defined in ways that prevent a researcher from knowing its 
true size, they have to be studied indirectly. This is usually done by 
drawing a sample of representatives (people) from that population, and 
making inferences about the population from the sampled evidence. The field 
of inferential statistics gives the groundrules for making generalizations 
about the population, based on evidence calculated from samples. 

For example, suppose we want to learn the success rate of Program X 
in dealing with the subpopulation of young, unemployed males. One could do 
such a study by taking a sample of the program's past young, unemployed 
males, determine the outcomes of interest, and calculate the success rate for 
that sample. One could then use that success rate to generalize to the 
population of young, unemployed males served by the program, witii an appro- 
priate **margin of error" that can be determined on the basis oT the sample 
size. One might also generalize to future samples of young, unemployed males, 
but only by assuming that the program doesn't change or that "young, un- 
employed males" do not change in important ways. 

Usually, program administrators and researchers would like to be 
able to make two different kinds of conclusive statements about outcomes in 
populations : 

1) They would like to know the outcomes for the entire population 

of clients served by a program, and for different subpopulations . 
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2) They would like to compare the outcomes for two or more 
subpopulations. For example, they might like to know if 
outcome rates are greater for women than for men. 

In order to make either of these kinds of conclusive statements about 
populations and subpopulations, one must first take appropriate samples of 
clients, observe the results for them and then draw conclusions, following 
the sort of "sampling process" outlined in the next section. 

A PROCESS FOR TAKING SAMPLES OF CLIENTS 

In order to make conclusions about populations of clients served by 

a program, a process for sampling clients must be followed. This sampling 

process involves several steps: 

Step 1: Specify the subpopulations for xvhich there is interest 
in making outcome statements, and define the population that 

includes all these subpopulations. 

Step 2: Specify how representatives of those subpopulations can 
be identified, where they could be found, and how their cooperation could 
be obtained. 

Step 3: Draw a sample from the population you defined, on whom 
data will be collected. 

Step A: Collect the data (by interview or questionnaire), and 
tabulate the results for each person in the total sample. 

Step 5: Group the data according to the subpopulations you are 
interested in, tabulate the results for each sample, and calculate the 
appropriate statistical test (discussed in Chapter XIII) that allows you to 
calculate the "margin of error" in your results. 
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Step 1: Define the population and subpopulatlons of in::ere st. 



Typically, in a program's first effort to assess proj^ram outromos, 
it will sample from the entire population of those who maclu contnct with 
the pro,^ram. Practically speaking, the largest sample one could draw from 
the population would be those people for whom a client record was completed 
and filed. 

Programs would probably define populations differently, depending 
on their services. For example, the nature of a client contact with an 
information and referral agency is different from a contact with a residential 
treatment program. These two kinds of programs would probably keep different 
kinds of records, so that the populations to be studied would be defined 
differently. An I&R agency might study the population of all those provided 
a referral. A residential treatment program might study the population of 
all those admitted to the program. 

At this stage, one should list the various subpopulatlons for whom 
outcomes will be assessed . This list should include any special "target groups*' 
the program intends to serve, whether defined by demographic data, chemical 
use, life situation, or program service history. The list should also 
include those groups that could serve as a basis for comparison. For example, 
outcomes for people who quit the program after the first day may or may not 
be of central interest, but knowledge of their outcomes certainly helps to 
give meaning, by comparison, to the outcomes attained by clients who do 
finish the program. 

Step 2; Specify how representatives of those populations can be identified 



and found. 
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For each subpopulation to be studied, there must be a way to recognizi 
individual members of that group. If you want to make outcome statements 
about "young, unemployed males" there must be a way to identify the age, 
employment status, and sex of each client. Usually this identification 
comes from the client's file, or from the follow-up contact itself. 

Also, there must be a way to locate members of those subpopulations 
you are interested in. Tf there is no locator information (address, phone 
number) then that subpopulation will not be well-represented in the study. 

Step 3; Select a sample of people 

Kow large a sample should be selected? For each subpopulation that 
you want to make outcome statements about, an adequate sample size must be 
chosen. It might be difficult to get adequate samples of complex sub- 
populations, such as the subpopulation of alcohol-abusing, f irst-tlme-in- 
treatment white males with intact employment and family. 

The rules of statistics tell us that the larger the sample, the 
better. This is simply because larger samples give more evidence of the 
population than do smaller samples. In other words, the accuracy of our 
knowledge of the population improves with evidence taken from larger samples. 
The average height of a population can be estimated with a smaller "margin 
of error" if the calculation is based on a sample of 300 than a sample of 30. 
While larger samples always make for smaller margins of error, a point of 
diminishing returns is noticed very quickly by statisticians. Very precise 
estimates can be made with samples of size 120. Satisfactory estimates can 
be made with samples of size 30. But keep in mind that smaller samples 
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always make for larger margins of error, so if it i important to bo 
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If one wants to know about outcomes in the population of people in 
contact with Program X, data from a sample of size 30 could give a reasonable 
estimate. But, if one wants to break down that population into various 
subpopulations , 30 is probably not enough. In other words, 30 is needed to 
give a satisfactory estimate for each population or subpopulation one is 
interested in. With reliable data, even smaller samples are tolerable. 

The size of the sample one can select is obviously limited by the 
number of people seen by the program in the past. This number does not 
determine or even suggest how many people should be In the sample, it only 
determines how many can be in the sample. The number that should be in the 
sample is governed by the number and variety of subpopulations that you want 
to study, and the margin of error that is tolerable in the results. 

Obviously, the larger the number of clients seen in the past, the 
greater the opportunity for studying a variety of subpopulations. If the 
program has seen only 30 clients, it is unlikely that there will be even 10 
alcohol-abusing, f irst-time-in-treatment white males with intact family and 
employment. With only 30 clients to draw from, very few subpopulations are 
likely to be found, and outcome results might best be expressed only for the 
total population. 

Suppose the program has seen large numbers of people, and the study 
design requires that only a portion of these is needed. How should these 
people be chosen? By far the most preferred procedure is to select people 
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randomly. A random selection process is one where each eligible person 
has an equally likely chance of being chosen. This could be accomplished 
by putting everyone's name into a hat, and choosing the required number while 
blindfolded. Another option is to consult a "Table of Random Numbers", 
found in most introductory statistics textbooks. 

A random selection process is the kind that will least likely yield 
a biased sample. If everyone has an equal chance being chosen, then whatever 
characteristics or qualities exist in the population (including factors that 
we don't have the foresight to know about) will probably appear in the sample 
to the same degree they appear in the population. This is especially true if 
the sample is relatively large. 

A large, randomly chosen sample of the population of clients served 
by a program could inform us of which subpopulations actually are represented 
in the program's clientele, and to what degree. This is important knowledge, 
even without collecting outcome data for each of the subpopulations. 

It is important to realize that the program will probably not be 
able to collect data on all the people in the chosen sample. Some people 
are not locatable, others will refuse to participate and some questionnaires 
may be returned in unusable form. One should draw an initial sample that is 
somewhat larger than what is needed, to allow for this attrition. 

Some of the reasons for attrition can introduce bias into the results. 
If people who are unlocatable or who refuse to participate are substantially 
different from those who are successfully followed-up, then there is a 
certain bias in the results. Even though the initial selection of the sample 
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followed a random process, the final ccllcM-tlon of data m;iy contain 
biases. The final sample is always of a population of pooplo who couid 
be found, and who were willing to participate in the study. 

If the completion rate of the sample if "low** (an arbitrary figure 
might be 80%), one should try to determine in what ways the respondents 
were different from the non-respondents. One should keep records, during 
the course of the study, of which members of the sample were unlocatable, 
and which refused to participate. Comparisons could be made of these 
different subgroups that might inform us of the biases contained in the final 
sample. If, for example, those who refuse to participate in the study 
contain a disproportionate number of people who left withcut approval, we 
might learn that the study sample is biased in a pro-program direction. 

Step A: Collect the data 

Procedures for interviewing, and recording the data will be discussed 
in subsequent chapters. 

Each answer in a person's interview must now be coded for analysis. 
Preceded questionnaires are the least time-consuming. Open-ended questions 
will require that categories be developed, and that each person's response 
be coded according to onp, of the categ-^'^ies. Considerable labor can be 
saved if categories are deveJopeu on the basis of pretest results, and then 
printed on the final version of the questionnaire. 

Step 5: Group the respondents, calculate outcomes, and the ''margin or error" 

The data must be grouped according to the subpopulation of interest, 
and outcome rates calculated for each. Data processing and statist^.^al issues 
will be discussed in later chapters. 
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Since a result or ouCcome rate Ik probably based on limited information 
(the sample rather than the entire population), it is likely to be in error 
to some extent. The size of this error, called by statisticians the 
"standard error of estimate", will depend mostly on the size of the sample, 
and partly on the variability within the sample. Larger sample size makes 
for smaller error, and larger variability within the sample makes for larger 
error. This will be discussed more fully in Chapter VIII. 

MAKING STATEMENTS ABOUT EFFECTIVENESS 

The term "effectiveness" is very broad, and means different things 
to different people. Tiilereut administrators give different definitions, 
and different evaluation resear :heb produce different evaluation designs. 
There are, apparent^/, v.Tany ways to offer evidence on "the effectiveness" 
of a program. 

The Handbook foas so far discussed two of the major ingredients 
for a systematic look at program effectiveness: outcome measures (Chapter II) 
and client sampling (in this chapter). The way different subpopulations 
are chosen for study, and compared to each other on outcomes, determines a 
goud deal about what can be concluded about program ef f-activeness . 

The kind of follow-up study emphasized so far in this Handbook leads 
to statements expressing the likelihood that each segment of the program's 
population would experience various desired outcomes. For example. If one 
found that 60% of the program's clientele were "not using intoxicating 
chemicals for one year following discharge", this translates into "the 
probability of this program's clientele not using intoxicating chemicals for 
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of Lhis outcronie for a variety of identifieiJ f^ubpopiilat i oiis In the program. 
One learns, in short, how well the program'^ clientele (and subgroups within 
that clientele) is likely to do, on a variety of criteria, based on how well 
it has done in the past. 

But knowledge of outcome rates for clients is not exactly the same 
as knowledge of the effect the program has on clients. We would not know, 
for example, outcome rates for people with the same health or pathology at 
admission but who receive different program services, or no program services. 
Since many people solve their chemical use problem without benefit of a 
program, it is important to know what difference it makes wl.ether one uses 
a program's services or not. It is exactly this difference that defines 
"the effect" of a program on a given population; at least, that is the way 
the term is used in scientific research. 

Answering the question scientifically, "Does this program have an 
effect?" is deceptively difficult to do. It ordinarily requires use of the 
classical experimental design, an ideal approach which is hardly ever seen 
outside of a laboratory or very controlled setting. This a-)proach requires 
a "treatment group" and a "control group." These two groups should be as 
alike as possible in all ways that could otherwise effect the results (same 
history, same present circumstances, same genes). If the two groups are 
entirely comparable, and if one group receives the program and the other group 
does not receive the program, then the difference in outcome between the two 
groups is an estimate of the impact of treatm-nnt on that population. 
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If the two groups are not entirely comparable, the estimate of the 
Impnct of treatment becomes more in error. If the i^roiips are vc^ry dissimilar 
in their pre-program makeup, differences in outcomes can possibly be explained 
better by these pre-existing differences than by program impact. 

In many ways, making a study scientific is at cross-purposes with 
making it practically useful. To answer the scientific question ("Does this 
program have an effect?") requires tremendous control and manipulation by 
the researcher, to make the comparison groups "equal enough" to measure an 
effect. It is highly worthwhile to know the outcomes of each major sc^mcint 
of the program's population, even if different segments cannot be properly 
compared with each other to produce an estimate of cause-and-ef f ec t . Program 
re-design is facilitated by knowing the outcomes for different groups of 
clients. 

Still, making comparisons is an important evaluation process, whether 
J-t's assessment of change, impact, or effectiveness. Knowing that an 
abstinence rate is 30% for females has greater meaning if we also discover 
that the abstinence rate for males is 60%. The magnitude of an outcome rate 
for a particular group of people takes on more meaning if we can compare it 
to another one. 

In a study where there are a variety of client subpopulations , a 
variety of treatment processes, a variety of outcomes of interest, and assess- 
ment at two or more points in time, there are several kinds of comparisons a 
program might want to make, all of which contribute to an 'unders tanding of 
program effectiveness. 

ERIC ^ 
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1) One could compare two kinds of clients on a particular outcome, 
The subpopuJ at ion of alcohol abusers could be compared to the subpopulation 
of barbiturate abusers on an outcome of non-use after one year, 

2) One could compare two kinds of treatment or two program components 
in their ability to yield a particular outcome. The subpopulation of those 
participating in aftercare could be compared to the subpopulation of those 
not participating in aftercare. Those leaving the program after 15 days 
could be compared with those leaving after 30, 

3) One could compare a single subpopulation against two different 
outcomes. The subpopulation of young males could be compared on an outcome 
of improved family relations and improved employment situation. With good 
intake data, one could learn which areas of peoples' lives are most impacted 
by program services, 

4) Various combinations of the above could be formed and compared. 
Young males leaving the program after 15 days could he compared with young 
males leaving the program after 30 days, 

5) One could make comparisons at two different points in time- Any 
of the types of groups above could be assessed at admission and dischnrj^e, 
admission and follow-up, or at two different occasions of follow-up. 

Furthermore, one is not at all limited to comparisons of only two 
groups at a time. Statistical procedures are available to compare any number 
of groups at a time on a given outcome. 

There are a few cautions to be observed in making group comparisons, 
and they concern the issue of the actual comparability of the groups. If 
we wsnt CO compare males and females on a particular outcome, we cannot say 
that differences in the outcome are "sex differences" unleso we kno^-' that the 
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two groups are alike in other important respects, if the group of males 
are primarily alcohol abusers, and the group of females are primarily 
tranquilizer abusers, then drug-of-choice is "confounded" with sex. Tha- 
is, the effect of drug-of-choice is inseparable from the effect of sex. 
One could "control for" drug-of-choice by making sure that the two groups 
have the same proportion of alcohol-to-tranquilizer abusers. This technique, 
called "matching," has the disadvantage of possibly creating other, 
unforeseen differences between the male and female groups. Possibly the 
best solution is not to claim "sex differences" in program effect, and instead 
to simply report the outcomes of the two groups. 

Another common example of pon-comparability of groups happens when 
one group is compared at two points in time, say at three months and six 
months after discharge. It often happens that the number of people successfully 
follow*=d-up at three months is considerably more than at six months. The 
6-month group could look exceptionally good if it's the "poor outcome" cases 
that drop out. In this case, it would be a serious mistake to conclude that 
"outcomes improve 3 to 6 months after discharge." 

C0NFIDENT1AI>ITY AND PRIVACY 

One of the factors affecting the availability of clients for follow- 
up is the rules and regulations governing the confidentiality of alcohol and 
drug records and the privacy of people. 

Regulation is needed to minimize the possibility of disclosure of 
potentially damaging information. Disclosure or misuse of information could 
result in embarrassment, lost opportunities, or other forms of psychological 
or social injury. The risk of these unfortunate consequences comes from the 
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of illegal drug use, criminal behavior, and feelings and values disclosed 
in the context of privacy and trust. 

Confidentiality and privacy guidelines and statutes govern access to 
records. Access to records is needed to get access to clients, since client 
locator information like address and phone number are contained in program 
records . 

Records also contain client description data and program service data, 
which are needed to describe client populations and subpopulations , to form 
comparison groups, or to form baseline data. 

Federal Regulations 

The Federal regulations are contained in "Confidentiality of Alcohol 
and Drug Abuse Records," originally published in the Federal Register, 
Volume 40, July 1, 1975, pages 27801-27821. They make up Part 2 of Title 42 
of the Code of Federal Regulations. A self-paced programmed instruction course 
on these regulations was published by the National Institute on Drug Abuse, 
and is available from the Chemical Dependency Programs Division of the Minnesota 
Department of Public Welfare. 

The following are highlights of the Federal regulations: 

They are concerned with alcohol and drug r^^-jords , and not any 
other kind of governmental or official records. 

They claim to present only the minimum requirements for the pro- 
tection of confidential records. 
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— They address the concerns of evaluation, by openly struggling 



with two streams of legal thought and social policy, one dealing with 
enhancing the quality of treatment opportunities, the otiier concerned with 
the interests of citizens as patients, most particularly in regard to 
protecting their rights of privacy. 

— They are supportive of research, audit, and evaluation, even 
without prior consent of the client: "...the content of records pertaining 
to any patient .may be disclosed, whether or not the patient gives consent, 
to qualified personnel for the purpose of conducting scientific research, 
management audits, financial audits, or program evaluation but such personnel 
may not identify, directly or indirectly, any individual patient in any report 
of such research, audit, or evaluation, or otherwise disclose patient 
identities in any manner." Getting informed prior consent of clients is not 
necessary, according to these Federal guidelines, since the risk of 
embarrassment or harm to the client from a follow-up contact is minimal 
(provided that only the program and client and not any third party is involved, 
deliberately or undeliberately) . 

— They defer to the State, when the State has privacy regulations 
that are more strict than the Federal regulations. Minnesota's Data Privacy 
Act ±s^ more strict, so it takes precedence. 

Minnesota Regulations 

There are two important Minnesota documents. One is the Minnesota 
Data Privacy Act, with 1978 amendments (Minnesota Statutes 15.162-15.1671), 
concerning "official records collection, security and dissemination of data." 
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The other is the proposed administrative rule, which if made official, 
defines terras used in the Act, establishes procedures concerning collection, 
use and dissemination of data on individuals, and defines duties of state 
and local officials who are responsible for data on individuals. Both 
documents are available from the Data Privacy Unit of the Minnesota Department 
of Administration. 

The following highlight the State's regulations: 

— They govern all official records, not just alcohol and drug records, 

— They are concerned primarily with specifying the groundrules by 
which official records are kept open or closed to the public, to the subjects 
of the records, and to government agencies. 

— They state that "private or confidential data on individuals shall 
not be used, collected, stored or disseminated for any purposes other than 
those stated to an individual at the time of collection...** 

— By an emergency action taken by the Commissioner of Administration, 
using a procedure made legal by the Data Privacy Act, all alcohol and drug 
records maintained by the Welfare system are classified as "private.'' This 
classification remains in effect until June 30, 1979, at which time it can 

be renewed. 

— "Private data on individuals" means data which are not accessible 

by the public, but which are accessible to the individual subject of that data. 

— "Data on individuals" includes all records, files and processes 
which contain any data in which an individual is or can be identified and 
which are retained or intended to be retained on a permanent or temporary basis. 

— The proposed rule also maintains that code numbers representing 
unique individuals in certain data constitute "data on individuals," provided 
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a list or Index Is made available by wlili-li the code number can be cross- 
referenced to identify unique individuals. 

— The law states that *'an individual asked to supply private or 
confidential data concerning himself shall be informed of: (a) the purpose 
and intended use of the requested data within the collecting state agency, 
political subdivision or statewide system; (b) whether he may refuse or is 
legally required to supply the requested data; (c) any known consequence 
arising from his supplying or refusing to supply private or confidential data; 
and (d) the identity of other persons or entities authorized by state or 
federal law to receive the data,*' 

The Privacy Act and the accompanying administrative rule suggest to us 
the following recommendations and observations: 

1) The program should inform the client at admission (or at any other, 
time when personally-identifying information is recorded) of all the ways that 
client data will be used, and who will see it. This applies to all uses of 
data, including those internal to the program (case management, referral, 
billing, mailing of newsletters). 

2) Getting a signed statement from the client to the effect he/she 
has been informed of his/her rights of privacy and the ways in which the 
program was ensuring it, is not required by the law, but it would give good 
evidence of compliance with the law. Another permissible, but less effective, 
practice might be to show that informing clients of their rights to privacy 

is standard program policy, perhaps by printing the information and guarantees 
of privacy on top of each form on which the client gives personally-identifying 
information. Questions of legal access to chemical dependency program clients 
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and client data should be directed to the Attorney Generars office in 
the State Department of Public Welfare. 

3) Informing the client at the beginning of the program (and 
reminding him/her again at the end) of the possibility that their name and 
address might be used to make contact for a follow-up interview is in 
keeping with the law. It has other advantages as well. It creates the 
expectation among clients that there will be a follow-up. It also provides ' 
an opportunity to ask for good client locator information. You could ask 
where the client could be reached in six months by phone, or bv mail. You 
could ask for ''safe" places to make contact with the client. It might be 
helpful to show the client the letterhead stationery on which the questionnaire 
(or any mail) might arrive, so the client will recognize it, especially if 
it would come from a source other than the program itself. 

A) Access to private files by third parties (research organizations, 
insurers, consultants, friends, and anyone else outside the client/program 
transaction) is illegal without the prior consent of the subjects of those 
files. It would have to be the program's staff that would make the first 
follow-up contact with the client. Staff could ask the client's cooperation 
in a follow-up study, and then schedule an interview with the program or a 
permitted third party. Or, the client could be asked to send in a form giving 
permission to be interviewed, and an addr'^iss or phone number to contact. 

5) Even if a client had not been told at admission of the possibility 
of a follow-up contact, it could be argued that the program has the right 
to make contact with its own former clients. Whether it is an invasion of 
privacy to make this contact has not been clearly stated. For a program to 
request a former client's participation in a follow-up study is not by itself 
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a betrayal of confidentiality, althouj?h tlie program must take care not 
to lunkr itself obvious to others In its searcfi for former clients. 

6) While It Is not at all clear from the regulations whether the 
follow-up interview itself is an "official record," it is probably best 

to regard it as "private data on individuals," even if a client's identity 
is not involved. This means that one must inform thp client, at follow-up, 
of the uses to which the follow-up data will be put, and \^ho will see it. 

7) Any program or administrative unit is permitted to provide 
summaries of data, and to make them public, even when derived from private 
data on individuals. These summaries, of course, must not reveal an 
individual 's identity. 

OTHER SOURCES OF DATA 

The discussion so far suggests that outcomes would be assessed by 
anayzing the self-report of the former client. There are two other major 
sources of data: report by a "significant other" to the client, and official 
records. 

"Significant Other" 

A major reason for collecting data from a significant other, or any 
other person known to the client, is to get corroboration of the client ^s 
story. It is often said that chemically dependent persons are apt to deny or 
at least distort certain facts about their lives. This could certainly 
introduce unreliability into the assessment of outcomes. 

For corroborative data to be valuable, we must be assured that the 
person chosen by the program to corroborate the client's story actually knows, 

Er|c 6^^ 
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and is willing to tell, the truth. For certain questions of fact this 
might bo true (such as whether the person Is employed, or receiving public 
assistance). But other, more personal ciucstions might not he accurately 
known to the corroborator. 

Recent evidence (McCrady and others, 1978) suggests that client's and 
spouse's reports of whether or not the client has been drinking tend to 
agree well, but there is disagreement over estimates of how much drinking. 

It is not clear how data from a significant other should be handled. 
If there is no agreement between client and corroborator how does one 
reconcile the two into a single summary? While consulting u significant 
other is no ciouht useful as aftercare, outreach, or intc^rvonL ion, it is n(^t 
clear what role it has in follow-up evaluation. 

Official records 

Some outcome area;; can be assessed from official records. One could 
iearn whether the per:>on is collecting unemployment insurance, registered 
with employment agencies, or caught up in the criminal justice system. There 
is an amazing amount of information kept in the records of government and 
the variety of "service delivery systems'* available to people. 

For these sources to be helpful, we must consider their accuracy, 
appropriateness, accessibility, and cost. The accuracy of almost all kinds 
of official records, even though they are "official", is surprisingly low. 
The most accurate public records are apparently birth records. 

The Sobell's (1978), who have done the most to document the reliability 
of various sources of data on treated alcoholics, conclude that the self-report 
of most clients is at least as accurate as most official records. 

67 
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Computerized data Is no guarantee of ncunirncy. iJ.-id data can 1h> 
made even wor«e when comp.1terl7.ed. ICvcn .sopl.l.sticntcMl computcrlxod data 
bases may contain considerable error just from mistakes in coding and 
keypunching. Other mistakes can occur from bad programming. 

Official records may not be altogether appropriate for use in assessing 
constructive change in behavior. Most records are kept by institutions that 
are concerned with different kinds of pathology or social deviance, like 
illness, criminal behavior, or poverty, rather than positive or constructive 
behavior. 

Official records might be useful for corroborating certain events, but 
one must be assured of the completeness of the records. Here too, disagreements 
between the client and the record present problems for analysis and summary 
of the data. 

Many official records may not be accessible to you, as governed by the 
Minnesota Data Privacy Act. 

The cost of collecting data from official records may be high, con- 
sidering the difficulties of securing permission and access, and the time 
spent attempting to locate all the possible places where your client's name 
or case might appear. 

Reliabili ty and validity of client's own self-report 

The Sobell's (1978) have concluded that the validity of alcoholics' 
self-report data is acceptably high. Some types of report are probably more 
valid than others. Validity of life history data is high, followed by "non- 
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alcoholic behavior data" and than "alcoholic behavior data" hut all three 
catej^orfes shov/ed enough vaJidity to proceed with evaluation rescarcii, 

Studies on the validity of self-repur ted cliemical u?e and chemical 
behavior often show conflicting results. Armor and his co-workers (1978) 
found studies to support most positions, and conclude that reliability and 
validity tend to vary according to the type of alcohol behavior, the type 
of respondert, and the type of setting. 

Si? :^ the respondent's need to deny or distort his/her self-report 
may be in part due to the stress or demands uZ the interview situation, 
there is a great deal the follow-up study designer and the interviewer can 
do to minimize th ^ threatening aspects of the follow-up transaction. 

This means giving attention to a) the way the program develops the 
expectation among clients that a follow-up contact is forthcoming, b) the way 
the program locates, or attempts to locate, former clients, c) the manner 
of seeking cooperation frcrr. the client, d) the assurances given to the client 
when informing him/her of rights to privacy, e) tindinf* a safe setting to 
conduct the interview, f) the content and wording of the questions, and 
g) the style of ihe interviewer. These will be discussed in the following 
three chapters. 
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CHAPTER IV. PLANNING A FCLLOW-UP STUDY: PROCEDURAL AND COST CONSIDERATIONS 



Once the objectives of a follow-up study have been determined, a 
program will need to examine the procedural options that are available. 
Finding the most effective and economical method for conducting follow-up 
involves a number of important considerations. 

This chapter will consider several procedural arrangements and 
explore the implications of each with regard to the following issues: 



1) ...Comprehensiveness of the study 

2) ...Personnel requirements 

3) . . .Confidentiality precautions 

4) ...Financial rf:sources 

5) . . .Consultation 



The decisions to be made vary in complexity. They range from such 
questions as who should collect the follow-up data to what kind of data 
collection methods should be used. The pui^pose of this chapter is to pro- 
vide a general introduction to these questions and show how each decision 
can effect the others. 

The chapter will also discuss some of the problems in achieving 
optimal credibility in the investigation of client outcomes and provide 
an overview of fundamental cost considerations. 



This chapter of "Evaluation Tools and Procedures" was prepared under 
Contract #29055-29056 with the State of Minnesota, Department of Public Welf< 



Judie J. Steiner, Steven E. Mayer, David S. Walonick 
Rainbow Research, Inc. 
Minneapolis, Minnesota 
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Preparatio n for follow-up 

A follow-up study for some programs might be premature. Programs 
which are very new or have made recent changes in their services may not 
have a large enough clientele to sample. For these programs, it may be 
helpful to begin preparing for future follow-up by designing a preliminary 
study plan using this chapter as a guideline. The preliminary plan can 
suggest ways to improve client intake procedures and will facilitate a 
smoother transition into follow-up when it is more appropriate. Ideally, 
clients should be informed of plans for follow-up and give their permission 
to be contacted, prior to receiving services. At that time the client 
can also be asked for names, addresses and phone numbers of persons who 
will always know his/her address. This procedure may also increase the 
probability for including clients in a sample who drop out of the program 
or discontinue services prematurely. The procedure for presenting client 
permission statements will be discussed further in Chapter V. 

Any program contemplating follow-up should examine its past and 
present intake practices for compliance to confidentiality regulations 
and client privacy rights before it attempts to contact former clients. 
This will enable a program to advise clients of their rights to privacy 
during the follow-up contact, if they had not been previously informed. 
(See Chapter III for specific regulations.) 
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Analyzing the audience and the nee..: for comprehensiveness 

An analysis of who will actually use the findings and for what 
purpose is necessary at the outset of planning. There are many reasons 
for conducting follow-up, some requiring more comprehensiveness tinan others. 
When important decisions w-11 be based on the study findings, it is 
critical that the study is scientifically sound, (i.e., a random sample, a 
high completion rate, reliable and valid data, and careful analysis of the 
data). The need for rigorous research procedures is especially important 
when the study results will be used to determine such things as requests for 
funding, staff allocations, or selection and modification of treatment 
components. For example, if programs are responding to a mandate for con)- 
prehensive evaluation from funders or political supporters, a sound research 
design will need to be developed to provide an extensive report of outcomes. 

On the other hand, some programs may discover that they can achieve 
the same objectives: of a study by some other less expensive means. This 
may be the case when a prograrr^ intends to use the findings itself, to answer 
internal programming questions. For example, a treatment program might want 
to assess outcomes of clients who participated in aftercare services and 
may have a thorough record keeping system indicating outcome information for 
each client that has used the aftercare component. An analysis of this 
outcome data may be sufficient to answer the questions of a follow-up study 
and consequently save the program from considerable effort and expense of 
gathering new data. 

Another example might be a program that would like to know if more 
effort should be made to help clients find emp'Ioympnt. In such a case, the 
program decision-makers can define what evidence will be sufficient to answer 
this question. The evidence needed may not require an extensive questionnaire 
or a large sample of clients. 
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Procedures and Contract ual .ArrangeiHgnts 

The research plan for a follow-up study is discussed in Chapters 
il and III. This section will discuss a procediii^al plan needed to implement 
a study. 

The responsibilities and functions of a foUow-up study can be broken 
down into three major phases: 

1) .. .Planning and coordination 

2) . . .Data collection 

3) ...Analysis and reporting 

Programs may conduct the folloW-Up study themselves through all three 
phases or contract with an evaluation/research organization to conduct the 
study. There is also the option of the Program providing a substantial part 
of the work force and contracting with consultants or hiring individuals to 
do specific functions where more expertise and/or objectivity is needed. 
There are advantages and disadvantages to each of these approaches, as 
discussed below. 

Developing a procedural ^ ajl 

The direction of this section is based on the premise that there will 
be a single follow-up of past clients of a program. The issues discussed 
will apply in most respects to other foUow-up alternatives; however, more 
complicated strategies, such as ongoing studies will not be addressed specifically. 

Assuming a program is ready to look at the procedural options, the first 
step is to begin to develop a plan for implementing the project, which will 
include a timetable and budget. The plan will influence the decision whether 
to do the project within the program independently, whether to have it done 
by professional researchers, or to select consultants at certain intervals 
9^. throughout the project. r*^ 
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It is helpful if one person is appointed as director of the study 
and be given cne responsibility to begin drafting the procedural plan. 
Experience in evaluation and research is desirable for this position, but 
at a minimum, the person should have an interest in research, a commitinent 
to a quality product, the time to devote to the project, and the willingness 
to learn evaluation methods from manuals or experienced researchers. 

In specific terms, the plan will first consist of defining the study 
objectives, the audience of the study, the sample and a description of what 
variables will be measured. The next consideration will be given to methods 
of data collection and selection of appropriate personnel to perform that 
function. The third phase concerning data analysis and reporting will be 
discussed in later chapters. 

Follow-up studies vary to such a great extent in goals, methods, and 
scope, that it is difficult to be specific about costs. The following 
suggestions are made based on our own and other researchers' experience. 

DATA COLLECTION METHODS 

Telephone Interviews 

The advantages of telephone interviews are that they are generally 
faster and less expensive. If the calls to clients will be local, there 
may be only the cost of additional lines or extension phones to be considered 
However, phone interviews should not exceed 20 minutes in length (10 minutes 
is better) and will require r?. simpler and shorter questionnaire in order to 
prevent the client from losing interest or becoming distracted in the process 
Time constraints with phone interviews will not allow the type of in-depth 
questioning that can be afforded in a face-to-face interview. 
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Tracing clients who have been out of the program for a long time will 
probably result in some out-of-state calls which would require additional 
long distance cost estimates. If the program clientele is highly transient, 
a phone interviewing procedure is not likely to produce a high completion 
rate of the sample. Also, the possibility that some former clients will 
not have phones will cause them to be lost from the sample, which could bias 
the findings of the study. 

Confidentiality is difficult to insure with telephone interviews and 
therefore, steps must be taken to prevent information regarding a client's 
participation in a program from being disclosed to others who may not know. 
Procedures for protecting client privacy with phone contacts will be 
discussed further in later chapters. 

Mailed Questionnaires 

While mailed questionnaires are often considered the least expensive 
method for getting information from clients, there are greater costs involved 
than one would imagine. 

The first aspect of additional cost is in the design of an effective 
questionnaire. The clarity, simplicity and attractiveness required of a 
mailed questionnaire will be enhanced by the knowledge and skill of a 
professional researcher early in the designing process. Consultation :uld 
be sought by a program if a mailed instrument is chosen for use in follow-up. 

This method will also require several mailings to those clients who do 
not respond to the initial request to return the questionnaire. The process 
of eliciting responses may take as many as 3 or 4 mailings to each non- 
respondent and will require a questionnaire with each reminder. This means 
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there will be additional expense for printing two to three times as many 
questionnaires and a sufficient quantity of persuasive reminder letters. 
It vn^] also effect postage expense appreciably, as each questionnaire must 
have a stamped return envelope accompanying it.* Some of the expense can be 
reduced if copies of all the follow-up documents are printed generously in 
the beginning of the project, through quantity printing price breaks. 

There are additional clerical costs for monitoring returns and mailing 
reminders persistently to non-responders . In terms of time-loss, delays 
are bound to be frequent in the completion of a mailed survey. 

Many mail follow-up studies produce less than satisfactory return rates, 
but this is usually due to the lack of design knowledge and rigorous follow- 
through procedures that are necessary for better results. 

Taking into account that a mailed instrument is one-way communication, 
there may be some problems with the analysis of the collected data. Even 
if designed well to permit the least amount of misunderstanding, questionnaire 
items can still be incorrectly answered or inadvertantly missed by respondents 
The occurrence of missed items, in combination with a poor response rate, 
could cause some analysis to be impossible, or at least inadequate for 
drawing any inferences from the collected data. 

Careful consideration must be given to mailed questionnaires to assure 
confidentiality. Maintaining privacy through the mail suggests that no 
reference is made to alcohol or other drug treatment, either in the return 
address on the envelope or in the content of the initial correspondence to 
clients asking for the participation in the study. A neutral agency name, 
or neutral research project title should be used to identify the follow-up 
study. Samples of introductory letters will be furnished in later chapers. 

♦Previous research has shown that the return rate of mailed questionnaires 
increases dramatically when a self-addressed stamped envelope is enclosed with 
the questionnaires. 
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In-person interviews 

Face-to-face interviewing has several advantages: it permits better 
confidentiality and privacy; it gives assurance of interviewing the right 
person; it allows more in-depth questioning; it facilitates clarification 
when questions or answers are unclear; and it permits more elaboration of 
answers from resf^ondents. Hov/ever, it is the most costly method of collecting 
da^a in terms of time and money. The actual expense will depend on the size 
of the sample, the amount of time to conduct the study, the ease in locating 
clients, the length of the interview, the extent of training provided to 
interviewers and the compensation agraements between interviewers and programs. 

Travel expense . The interviewer will need a car and mileage allowance to 
v'acilitate interviewing those clients who do not have transportation or 
prefer to be interviewed in their homes. If the program serves a rural area 
or if clients have moved out of the locale of the agency, travel costs will 
be somewhat greater. 

Administrative and office expense . If clients are asked to meet with the 
interviewer at the program or some other office, consideration needs to be 
gi/'en to providing these facilities. Office space for contacting and inter- 
viewing clients should include phone, easy accessibility, privacy and a 
neutral atmosphere. 

If a program decides to hire interviewers, salary and othar benefits such 
as insurance, tax withholding and overtime must be negotiated. Paying an 
interviewer by "completed interview" is complicated by issues of client 
accessibility and quality control, and is not recommended. Well supervised 
interviewers will work as efficiently on a salary or hourly rate. 
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The responsibility of recruiting, training and supervising inter- 
viewers, regardless of their number, needs careful attention. A program 
will want to delegate these responsibilities to one or more people who 
have ample time, sufficient skill and good accessibility throughout the 
project. The use of professional research organization may be seriously 
considered for performing these functions, particularly in large studies 
where the management of interviewers and data collection is time consuming 
and complicated. 

If the program will use its existing staff to perform the interviewing 
function, the major cosi will be the time its own staff devote to the project. 
But if the sample size is large and the time to complete the study is a 
constraining factor, there may be a need to bring in other people to assist 
in the interviewing. 

WHO SHOULD DO THE INTERVIEWING 

Another aspect of in-person interviews is the issue of selecting 
appropriate people to conduct them. Objectivity, availability, asserti veness 
and interviewing skills are the essential requirements for good interviewers. 
Beyond these basic requirements, there are other considerations for appro- 
priateness. Th? credibility of the study, confidentiality concerns, and cost 
are reasons why a program will need to examine the following discussion 
carefully, to determine if existing staff, hired personnel or a research 
organization can provide the best interviewers for the project. 

Using existing program staff 

Using existing program staff, such as counselors, has some advantages. 
In certain programs, it is likely that the staff would have more success in- 
locating former clients and scheduling an interview with them than would an 
outside researcher. ^T.O 
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If counselors of a program vera assigned to follow-up their clients, their 
knowledge of the clients' histories would also provide a check on the infor- 
mation that is given. This type of validity check would not be available 
to outside interviewers, unless history information was provided to them 
in summary prior to the interview. 

In addition Lo staff's familiarity with clients, they would also have 
prior knowledge of the record keeping system and generally will have less 
trouble interpreting and transferring correct information on clients. If 
hired intervievyers are used, it will be necessary to provide orientation to 
familiarize them with the record keeping system, client population and 
philosophy of the program. 

Using program staff has the main advantage of ease and simplicity in 
dealing with confidentiality requirements. These requirements make follow- 
up less complicated when only two parties are involved, the program and the 
client. Essentially, the program has the right to access its own files to 
compile client locator information and contact clients to request their 
participation in follow-up. A third party, such as an employee of a research 
organization, does not have the right to access the program's client files 
wi uhout the prior consent of the client.* 

The job of retrieving information from the files, updating address\s, 
and locating clients can probably be done less expensively by the existing 
program staff. If previous client consent has not been obtained for follow- 
up, it will be necessary to use employees of the program for the initial 
contact to obtain permission. This may b''- done by mail or phone. 

*There is some question as to who constitutes a "third party". For example, 
a program managed ent-irely by an Area Board (as a fiscal affiliate) is con- 
tractually linked. to that Board. Therefore, a Board employee apparently has the 
right to access the program's files. It-is highly advisable for a program to 
get an opinion or consultation from the State's Attorney General attached to 
the Department of Public Welfare, regarding the "Minnesota Data Privacy Act", 
if it is in doubt about access to program files. 
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The use of existing program staff as interviewers has several dis- 
advantages that need consideration. 

The first consideration is whether former clients would be as honest 
with staff interviewers as they would with someone they did not know or 
associate with the program prior to the interview. There is the distinct 
possibility that clients will wish to please staff interviewers that they 
know and who may have had an impact on their lives. This coula lead to 
positive responses that are influenced more by the situation than by tne 
truth. 

The second consideration is the potential feeling of threat to staff. 
If the study findings will be used to determine directly or indirectly the 
future staffing of a program or effect the program's reputation, it is 
unrealistic to expect the program staff to be fully objective in collecting 
data. 

Beyond the fear of losing one's job, there are more subtle th. eats 
that may affect staff objectivity az interviewers. Most therapists or 
counselors are convinced there is value in the work they do, in spite of 
the frustrations and disappointments that they face on a daily basis. They 
may find it difficult to accept that not as many people were helped as they 
had hoped. Therapists also risk learning that former clients have been 
helped more effectively by some other treatment or no treatment, which may 
cause them to feel that they have been wasting their time as "helpers". 

In many programs there may be no risk of deliberate distortion of 
the findings by its staff, but there will always be the risk of unconscious 
or seemingly innocent efforts to manipulate the findings. This unconscious 
bias can manifest itself in a number of ways. S.aff could be selective 
in their efforts to contact clients, putting more energy into locating 
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those clients that will give positive outcomes and/or passively dealing 
with those clients that may be negative towards the program. As a 
result, the interviewed sample of the program's clientele will be biased. 

Bias may also surface in the interview itself, particularly if the 
staff interviewer records responses to open-ended questions. It is likely 
that more elaboration will be elicited and recorded for positive comments 
than negative comments. 

When an interviewer stands to gain or lose from the findings, for 
whatever reason, the question should be raised whether that person is 
appropriate to collect data. The use of outside interviewers will maximize 
objectivity and should be given careful consideration. 

Furthermore, if a former client feels some threat of repercussion 
from a program to negative comments, he/she will not want to share them 
with a staff interviewer that may expose his/her identity. The use of an 
outside interviewer would allow former clients to feel more certain that 
their identity will not be learned by the program. 

Another disadvantage is that, even though staff are often very willing 
to help in evaluation, their availability for contacting and interviewing 
clients in a follow-up project may be difficult to arrange and coordinate. 
If staff are utilized to a maximum capacity for program duties and then 
expected to take on the additional job of follow-up, it is likely the 
additional tasks will be executed poorly, inconsistantly , or not at all. 
There may also be implications of poor staff morale under this type of 
pressure. 



Minimizing risks of bias 

If it is not possible to hire outside interviewers due to cost or 
lack of resources, there are some ways to minimize these risks of 
subjectivity. 

The interview itself may be designed to facilitate more objective 
information gathering (e.g., more structured, closed-ended questions), thus 
making it less susceptible to interviewer bias. However, even with a care- 
fully designed questionnaire, the delivery of the same question, depending 
on the tone o. voice or non-verbal cues, can elicit quite different responses. 

Another way of minimizing the risks of subjectivity and over- 
extension of staff would be to select personnel whc have had the least 
contact with clients, possibly the administrative staff, interns or volunteers 
These people may be more easily rescheduled to allow for the additional work. 
Candidates for the role of interviewer should be examined to see who has 
the least stake in the findings, both personally and professionally. 

The nature of the audience will also be a consideration. If data 
are being collected for a funding agency or legislative body, the use of 
the program's own staff as collectors becomes more questionable. It is 
likely that the public and funders will have greater confidence in the 
objectivity of an independent researcher than they would in the program's 
own assessment of outcomes. 

Hiring additional staff as interviewers % 

Hiring a person outside of the program is a compromise between using 
the existing staff and contracting with a research organization to conduct 
the interviews. This arrangement has the advantage of complying simply 
with the confidentiality regulations, since the new staff person would have 
legal access to client files, 

EKLC 



IV-14 



The major advantage is that a less biased person could be hired to 
do the data collection and is apt to do the job more effective!- ^mp to 
a clearly defined role within the program. 

This arrangement may have the disadvantage of former clients perceiving 
the interviewer as an agent of the program, which could affect the client's 
decision to be candid. However, the association would have less influence 
on a client than would his/her former counselor. 

Interviewer training will be necessary regardless of the arrangements . 
The extent end focus of this training will depend a great deal on the 
comprehensiveness of the study and the qualifications of the interviewers. 

There is also the issue of cost; hiring an interviewer will obviously 
mean adding him/her to the program's payroll- 

Contractinq for interviewers through a research organ ize tign 

A professional research organization r 9 (untracted to conduct 
the studv and/or provide interviewers. Ih' » arrangement would present the 
greatest possibility for objectivity, since t.'ie interviewers would not 
have a stake in the findings. Former clients may also respond more 
accurately to these interviewers who are seen as "neutral", thus lending 
more credibility to the findings- 

One of the major disadvantages in thi'. type of arrangement is that 
it will be the most expensive approach. 

Another disadvantage s the problem of access to records for a third 
party. One solution, as mentioned earlier, is to inform clients at admission 
of the follow-up project and obtain permission from them to be contacted 
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by the research group. However, this must be done well in advance of the 
study and there is still the likelihood that not all the clients will 
give permission at intake. 

Another solution is for program personnel to make the initial contact 
with clients requesting their participation in follow-up. While this 
arrangement falls within the confidentiality guidelines, it could lead to 
a biased sample, if the program is selective of the people to be interviewed. 
The research group under contract should have the authority to monitor the 
random sampling procedure, without jeopardizing confidentiality. 
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UEVELO PING A TIMETABLE . AND AUjJ_GiJ. 

After consideration has been given to methods and personnel, the 
procedural plan will need to contain estimates of time and costs in the 
form of a timetable and budget. 

Planning a timetable 

It is essential that a written plan is used as a guide to developing 
a timetable, stating target dates for each phase of the study. The following 
tasks must be included in the time estimates.* 

1) ...Selecting the sample 

2) ...Developing the questionnaire 

3) ...Training interviewers 

4) ...Compiling locator information 

5) ...Conducting a pilot test 

6) ...Locating clients and scheduling interviews 

7) ...Conducting interviews 

8) ...Editing and coding collected data 

9) ...Keypunching and verifying 

10) . . .Analyzing the data 

11) ...Preparing the report 

*The above list is applicable to studies using in-person interviews 
and computer analysis. 

A time estimatefor each phase, 3ven if inaccurate, will have value for 
a number of reasons. It will allow the program to estimate how long it 
will take to complete the project, it will lead to a list of all the tasks 
that are to be performed, and will provide a base for discussion on how the 
tasks will be executed. It will also highlight the areas where consultation 
and special training will be needed. 

^6 
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Once the project is underway, the timetable will help the project 
director identify problems when delays arise and give assistance in 
keeping the study on track. A good rule of thumb is that each step will 
require more time than one would expect. 

Locating clients . The time it takes to do the actual interviewing will 
usually be much less than the time required to locate clients and arrange 
for the interview. A pilot study will provide a base for time estimates 
related to tracking clients. (Chapter V will discuss the procedure of a 
pilot study. ) 

If a completion rate of 85% or more of the sample is sought, the 
methods lur locating clients become painstakingly thorough. It will obviously 
take longer to locate those clients who have moved. The time requirement 
for locating clients will be correspondingly greater for each year the client 
has been out of the program. 

Planning a budget 

The greatest expense of a follow-up study will be absorbed by personnel 
regardless of the contractual options chosen. A breakdown of all the tasks 
to be performed and tha length of time allowed for each task will give 
structure to the budget in terms of person-hours required. 

If the program's existing staff is used in the project, even without 
additional payment for their services, it will involve hidden costs. Diverting 
staff from their .egular duties will mean the program will have to decide how 
those duties will be taken care of in their absence. 

An evaluation of person-hours may lead to the realization that the 
follow-up project is too ambitious and v/ill require a scaling-down due to 
Q personnel shortages. It is much better if this scaling down occurs in the 
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planning stages of a study, as opposed to drastically reducing efforts or 
abando.-.ing the study later in the project. An untimely decision to reduce 
efforts would obviously lead to considerable waste and/or reduce the use- 
fulness of the study findings. 

Other follow-up expense 

C lerical costs . Maintaining records on attempts to find and interview 
clients, and to edit the collected data, take a great deal of time. These 
tasks may be shared by the coordinator, staff and interviewers, or may be 
the interviewers responsibility alone. 

Paying respondents . Many studies have successfully included payments to 
clients for granting an interview. A general rule, however, is to make the 
amount small enough to be feasible and yet large enough to give the client a 
sense of responsibility or obligation to follow through with the interview. 
The range of payments could be anywhere from a quarter enclosed with a 
mailed questionnaire to $5.00 for an in-person interview. 

Another way to compensate clients, is to pay for their travel expense 
(cab, bus, or mileage) when they agree to meet at the interviewer* s office 
for an in-person interview. 

For some programs, it may seem extravagant to consider this type of 
payment, but a high completion rate of the sample will be of great importance 
If not achieved, it could significantly reduce the scientific value of the 
findings. 

Table 1, on the next page, outlines various cost items which must be con 
sidered when constructing a budget. When the best possible estimates have 
been made, the program should be aware that upforseen events may increase the 
costs. The study should not be approved unless there are sufficient funds to 

cover the entire project. SQ 



TABLE 1 

Personnel Expense 

Research director/coordinator 

Staff or other personnel for: 

Search of agency records 
Contacting clients 
Interviewing 

Monitoring oata collection 
Coding collected data 
Keypunching 
Data processing 

Final report production 
Secretarial support 

Consul tants 

Materials and Interviewing Expense 

Printing and duplicating: 

Questionnaires 
Stationery, post-cards 
Return envelopes (if any) 
Xeroxing (letters, tables, etc.) 
Final report 

Postage 
Telephone 
Computer time 
Office supplies 

Travel expense (Interviewers and/or clients) 
Payment to respondents (if any) 
Office space (if necessary) 
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Adj usting th e study to resource s availab'ie. The value of the expected 
findings should be weighed carefully once the cost of the study is estimated. 
When the evaluation funds are not adequate to support the original study 
plan, revisions may b'* made to reduce the study objectives to a more 
specific area of effectiveness, and/or a smaller sample can be drawn. 

Resources for follow-up . Evaluation monies are sometimes available through 
the Minnesota Department of Public Welfare, Area Mental Health Boards , or 
foundations. 

Contracting with outside research groups 

If an outside research group is chosen, there may be several advantages 
in joining with one or more similar programs in the area for the purchase 
of research services. The costs of developing interviews and procedures 
could be shared by the programs, and assuming they are compatible modalities, 
the programs would have other outcome data with which to compare their own results. 

When contracting the entire study to an outside research organization, 
a written contract must be established. Such a contract must specify the 
costs, time schedule, responsibilities of each party, ownership of the data, 
and the final product. It should also be stated that the questionnaires, 
computer cards and data tapes be turned over to the program when the contract 
is completed. This way a program would have the capability to reanalyze 
the data or perform further analysis at a later date. If the same clients 
are to be followed-up at some point in the future, the program will alio want 
copies of the locating and interviewing records, with the key so that clients 
can be recontacted. 

Another issue to be made clear at the outset are the publishing rights 
of the contractor, whether they may publish independently from the program's 
data and/or identify thci program with the findings. 
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Consultants 

The use of consultants may be appropriate at almost any stage of the 
study. However, a day of a consultant's time can cost well over a hundred 
dollars and it behooves a program to get as much as possible from that time. 
A program can expect to get significant help from a consultant if its 
questions are thoughtfully formulated and specific. 

There is a point where consultation becomes especially important, and 
that is when the research plan has been developed in writing and is being 
considered for approval. A consultant can review the plan for flaws in 
design from a technical perspective. The review can also suggest how expenses 
can be reduced, how efficiency can be increased and how realistic the pro- 
jected costs appear. A consultant can also review the plan for legal or 
ethical problems with regard to confidentiality. The laws are quite complex 
and they are continually changing. The ethical requirements lend themselves 
to varied interpretations and unless a program is completely familiar with 
them, it is wise to get consultation in this area. 

If a program is considering a contract with a professional research organ- 
ization, consultation may be advisable for finding an appropriate contractor. 
There are special problems ass-^ciated with selection of a contractor, but 
the cost of a consultant for t.os will be minor compared to the cost of the 
contract itself. 

Another area where consultation is especially valuable is in the finai 
reporting stage of the study. Once a draft is prepared, an opinion from a 
professional researcher can be sought with regard to the analysis and the 
conclusions that are made in the report. 
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The number of evaluation specialists and researchers is increasing, 
and there are some who have had relevant experience in the area of chemical 
dependency programs or similar social -honl t:h service pro(|roiriS. Oth(M^s 
nay not be familiar with chemical dependency as such, but are exports in 
research design and have experience in studies of other populations. 

Information on potential consultants can be found in professional institutes 
for survey research, in universities, in some funding agencies, and in the 
Single State Agency's Planning, Research and Evaluation Section. 
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CHAPTER V. Planning A Follow-up Study: Operating Procedures^ 

by 

Judie J. Steiner, Steven E. Mayer, David S. Waionick 
Rainbow Research, Inc. 
Minneapolis, Minnesota 

Chapter IV discussed sevaral procedural arrangements for 
follow-up. Each procedural option involves two essential tasks: finding 
the subjects of the study, and gaining their cooperation to participate 
in the project. Since these tasks will absorb a large portion of the 
Lime, effort, and money that is spent on the project, it is important to 
develop a comprehensive strategy for this phase of the study. 

This chapter v;ill consider operational procedures of follow-up 
with regard to the following steps: 

1) — Compiling records for administering the study. 

2) ... Locating clients from available resources. 

3) ... Informing clients of the study to gain their cooperation. 
A) — Pretesting the study procedures. 

The next chapter will provide methods for coordinating and monitoring 
the data collection process and will discuss the qualifications and training 
of interviewers. 



"This chapter of "Evaluation Tools and Procedures" was prepared under 
Contract //29055-29056 with the State of Minnesota, Department of Publi 
Welfare. 
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DEVELOPING RECORDS FOR FOLLOW-UP 

• 

Reviewing agency records 

As mentioned in earlier chapters, one of the initial steps for 
follow-up is the careful review of the program's client records. This 
review is done early in the planning stages to determine what client 
information is available. 

An examination of the records will reveal the possible ways oT 
defining subgroups, and will also provide an estimace of the difficulty 
in locating clients from f;he addresses provided. While this examination 
will point out limitations for the present follow-up project, it may also 
lead to iiiprovements in record keeping for future studies. 

Administrative and research records 

Once the research plan has been approved, it will be necessary for 
the follow-up director/coordinator to develop several sets of records 
that are essential for effective study management. 

List of clients . The first document required is a complete list of all 
the clients who are eligible for the sample, with a written description 
of how eligibility is defined. 

The list of clients may be either by name or by program client 
number. It is from this list that a special study number will be given to 
each client. The numbers should begin with 001, 002, 003, and so on, using 
enough zeros so that all numbers on the list have the same number of 
digits. When all of the clients have been assigned a study number, the list 
of clients should be kept by the study coordinator to ensure its safety 
from loss or damage. This list is what is known as a "client identifier key." 
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The procedure of assigning new numbers to the potential study group 
permits better data privacy. All of the collected data should be identified 
by the client's study number, and separated from any personally 
identifying data. Only the staff contacting and interviewing the sample 
will know the clients' names. The special study number will also assist 
in the random selection process if a random number table is used. 

The same type of list is necessary for identifying control or comparison 
groups if these are used. A clear record should also be kept of exactly 
how each sample was drawn from its population. 

Consideration for repeaters in sampling procedure . Some programs may 

decide to sample from their entire list of admits for a given period of 

time in a follow-up study. This type of sample poses the problem of 

dealing with individuals who have been admitted to the program more than 

once during the designated time period. The objective will be to eliminate 

duplicate admits from the selected sample for follow-up. 

Procedures may vary from program to program, but generally a file 

is established the first time an individual is admitted to a program. Also, 

the admission is usually entered in a log by date, using either the client's 

name, or a client number assigned by the program. For clients who are 

assigned a number and readmitted into the program later, there is the 

possibility that they could have had more than one number entered in the 

records. As a result, there could be two files for one individual. This may 

not be a common occurrence, but programs should be awara that over a period 

of years, different personnel and procedures may have created inconsistencies 

and errors in client records. 

"^It has been determined from the administrative rule governing the 
"Minnesota Date Privacy Act" that a program's client number could identify 
an individual in some circumstances, 
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The first approach to compiling a list of eligible clients and 
identifying repeaters requires that the case files be checked against the 
log of admits- If two or more client files exist for the same person, 
they should be consolidated, so that one file includes all admittances. 
A new list of clients should be made by eliminating duplicate admits 
from the log, which will result in an accounting of how many individuals 
were actually seen by the program. 

For example, a program decides to sample from a population of 
former clients seen over the past two years. Thp records indicate that there 
were 325 admits during that period. After checking for repeaters, the 
program discovers that 50 admits were clients who were previously admitted 
in the two year period. The population would actually consist of 275 people. 
A study number would then be assigned to each client, thus assuring that 
the same person would only come up once when the sample is dravm. 

Another approach would be to assign each admit a study number before 
the sample is drawn. Then each number- would be checked against the client 
files to see if there were any duplications of individuals. If a duplicate 
is discovered, it would be discarded and a new number would be drawn until 
each individual in the Gample is represented only once. 

When eligibility for a study is determined by membership in a 
special subpopulation instead of by program participants from a specific 
time period, the list of clients will need to be compiled from the data in 
the case files. However, this method requires that each file be examined 
for consistency in the recording of information that is needed to define 
eligibility and will take considerably more time than the previously 
mentioned methods. 
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Master client file . After the sample has been drawn, it will be 
necessary to establish a card or folder for each client within the .sample, 
using the assigned study number. This will scrvv as a master client file 
to be used by follow-up staff throughout the study. 

The master file should contain information which will help locate a 
respondent and should also include a list of identifying characteristics. 
Items such as addresses, birth date, nicknames, occupations, social security 
number, natnes and addresses of relatives or friends who might know his/her 
current location, and any other descriptive data should be included on n 
"search sheet.'' This information will be used by those persons who 
attempt to locate the client, and by the interviewer to confirm that the 
respondent is the correct person. 

Another purpose of the master client file is to reveal at a glance 
what has been done and what remains to be done to complete the follow-up. 
At a minimum, the records should indicate the following: 1) what has been 
done to trace the client, the leads thar have been exhausted and the new 
leads that have been found; 2) the efforts made to interview the client, 
and their results; 3) location, method and date of the interview; 4) whether 
the data collected have been edited for completeness and clarity, including 
further contacts by the interviewer to get missing information; 5) whether 
the interview data have been turned over to analysis personnel or coded for 
analysis . 

Similarly, other steps for retrieving information from the client 
files should be listed and checked off in the master file. The forms used 
should provide enough space for dates to indicate when each step is completed, 
and to describe reasons for any difficulties and what is to be done about 
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them. In general, the simpler the forms, the better, but tliey must 
contain enough information so that the project coordinator can quLckly 
review the status of all cases. It will tlien be possible to make periodic 
checks to see if the project timetable is being met, and to identify any 
problem areas that need attention. 

Documentation of decisions . Research events and decisions should be 
recorded as they occur throughout the entire research process. There will 
undoubtedly be a significant number of emerging problems and decisions to 
be made during the study, and it is amazing how quickly one can forget 
which decisions were made and why. A careful record of these decisions 
should be preserved, since a detailed accounting of procedures will enliance 
the meaning of the findings. 
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LOCATINC CLIENTS 

The study coordinator will need to anticip;it;e the ty!)es of 
problems that may arise in locating clients and scheduling interviews. 
Certain problems will be due to the specific nature of the program's 
services and the clientele it serves. Nevertheless, most programs that 
attempt to follow-up clients who have been out of the program for several 
years will find the task of locating a high percentage of the sample 
particularly challenging- . 

Problems iii locauing clients -' 

Certainly, chemical dependency programs serve a wide variety of 
people. However, there are some circumstances which may make the task of 
finding clients of chemical dependency programs more difficult than in a 
study of the general public. Certain clientele may represent an extreme 
case for the following reasons: 1) they are more mobile than the average 
follow-up subject; 2) they are unemployed and poor, which further increases 
their mobility; 3) they are Icisz often registered in many sources that 
provide changes of address for most people, such as the phone book, post 
office listings, voting records, etc.; A) they may have poor ability to 
sustain personal relationships, which means their relatives, spouses or 
earlier friends may no longer know where they are and may not have heard from 
them for a long time. Even if found, some may not respond to a follow-up 
contact due to their being engaged in illegal activity and/or illicit drug 
use, out of fear of arrest. 

"^This section is adapted from the Treatment Program Monograph Series. No. 2, 
Conducting Follow-Up Research on Drug Treatment Programs , Chapter 5, Ethical 
Considerations , written by Lee N. Robins and Edward J. Smith, published 
by NIDA, 1977. 
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Some clients will be a great deal easier to locate than others. 
Those who have remained in a program longer will be found more easily 
because? Lh(?Ir habits and interests will be better known and their records 
will contain more clues for locating them. 

Further, those clients who are doing well, whether as a result of 
treatment or spontaneously, are more likely to have a life-style which 
involves less moving, better emplojnnent status, more contact with relatives 
and friends, more community exposure, and consequent ease of location. 

At the same time, some of those who did not do well after the 
program may be back in treatment ^ incarcerated, or may have died. These 
outcomes may be documented through examination of of icial records or 
through contact with relatives or friends. However, it is important that 
the status of each client is verified as thoroughly as possible before it 
is defrermined that he/she is unavailable for an interview. 

Since ease of location is usf.rally associated with positive outcomes, 
recovery of a high percentage of the sample is necessary to avoid having 
a biased estimate of outcomes in the study's findings. 

Consideration of cost 

For the researcher, the matter of hard-to-locate or reluctant clients 
raises the question of when to give up. This question is related to the 
study's cost-effectiveness. Unavailability and refusal to participate is 
inevitable for some clients, but a study with a large number of completed 
interviews will give a more realistic representation of program clientele. 
In lost studies, interviews on the firsc half of the sample will be relatively 
inexpensive; the next 15-20 percent will cost appreciably more. Completing 
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the last 20% could cost twice as much per case as early interviews. 
Consequently, a study with a high completion rate cannot be done cheaply. 

Resources for locating clients 

Public Records. City directories and telephone listings are obvious 
sources for locating clients and contacting relatives and friends to learn 
the client's whereabouts. If this fails, a street address directory can 
be rented from the telephone company. This directory lists telephone number 
by residence instead of name and it allows one to call former neighbors of 
a client to get information on where the client: may have gone. 

Another major source of help is the U.S. Post Office, which records 
address changes. For about 25c per address, the post office will correct 
names and addresses if there is a change notice on file, and when "address 
correction requested" is printed on the envelope. The "special delivery/ 
return receipt requested" service is also helpful because the name of the 
person who accepted the letter will be supplied and this may provide an 
additional lead to the client. In addition, a special delivery letter 
emphasizes the importance of the study and increases the likelihood that 
the client will read it more carefully when it is received. 

Other agency sources (private records) > Certain agencies can be contacted 
if routine methods have proved unsuccessful. However, many of these 
agencies are now considered holders of private data on individuals and will 
require the client's written consent to release information before it can 
be given to researchers. As a result, these sources will be considerably 
more difficult to tap for client locator information. 

Some of the sources within the state that can provide information, 
if there is client consent, are law enforcement agencies (police, prison, 
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parole and probation records), the state department of health, social 
services or welfare agencies, state hospitals. Veterans Administration 
Hospitals, and other chemical dependency programs. 

Issues of confidentiality are of great concern here, and the 
recommendations discussed later in this chapter should be considered 
carefully, particularly those concerned with sharing information about a 
client beyond what is already known. 

Permission and consent forms 

There are several procedures for preparing clients for follow-up 
and eliciting clients' participation in a follow-up study. The following 
discussion concerns the appropriate methods for each of the procedures, 
so that they comply with confidentiality and privacy regulations. 

Obtaining a signed statement from clients that they will participate 
in a follow-up study is most advantageous if it is done well in advance of 
the study. This can be done either at admittance or discharge from the 
program. 

There are four major reasons for presenting permission forms: 
1) it advises the client of the study; 2) it provides an opportunity for 
the program to learn where a client will be at a later date; 3) it will 
ensure that the client has been informed of his/her confidentiality and 
privacy rights; ^) J.f specified, it will give a third party, such as a 
research group, the right to contact the client directly. 

Permission forms can also be used just prior to the interview, to 
document that the client understands the nature of his/her participation. 
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Below is a sample of a permission Form used in a recent study. 
It was presented to clients before they were discharged from the program. 



Permission Form 

Please read the following and sign your name below, if you agree to 

participate in a follow-up study of . 

(name of program) 

1. I understand that if I give my permission to be interviewed, 

I will be asked questions about my experience with the program, 
my chemical use and changes in my life since I was in the 
program. 

2. I understand that my participation is voluntary and that I may 
withdraw my permission at any time without negative consequence. 

3. I understand that all information is confidential and that my 
name or identity will not be used or revealed in any way. 

4. I understand that the purposes of the study are to find out about 
the services I received, my opinion about the program and to 
learn about the role of alcohol and/or other drugs in my life. 

I also understand that the study findings will be used to help 
plan better programs and services in the future. 



Client's Signature Date 

Thank you very much for your cooperation. If you agreed to participate, 
please give the following information. 

Present address 



Present phone number 

\Jhat time of day would be best to call?_ 

Name and address of person who will know how to reach you 



If not participating, please note reason. 



Form presented by: 

(Staff person) 



Date 
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The staff person who presents the form to a client should be prepared 
to answer any questions that the client may have and should also be 
responsible for putting the form in the client's file. If the client refuses 
to participate in the study, any reasons for the refusal should be noted 
as well. This way, a program will best be able to document those clients 
who agreed to participate and those who did not. Close monitoring of this 
procedure is essential, so that a client is not omitted from the process. 

The permission form is a variation of what is known as an "informed 
consent" statement; however, it speaks only to a client's participation in 
a follow-up study and does not address other authorizations for the program. 
Program authorization to obtain information about a client from other agencies 
or persons, and/or allow a program to release information to other specified 
persons or agencies requires a different type of written statement. It is 
necessary to state on these forms which agencies will be contacted or what 
records will be examined, and to give a date for which the authorization 
to search or disclose expires. For further information on statements of 
consent, a program should consult the "Minnesota Data Privacy Act" and 
confidentiality guidelines provided by the state. 

Other considerations for consent forms . Obtaining signed permission and 
consent forms at admission to a program can present both psychological and 
ethical problems.^ Requesting a client's agreement to these statements 
may cause a client to feel pressured to cooperate. The program's attempt 
to obtain permission may imply to them that they have to consent to follow- 
up in order to be admitted into the program. It is important to make clear 
to the client that receipt of services is not in any way contingent upon 
participation in follow-up. 

^Ibid. 
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Whenever a program successfully gains access to other records or 
gets authorization to contact persons for information regarding a client, 
two general operating principles should be followed. First, clients should 
never be identified to any person or agency as persons who have received 
drug-related services, unless this information is already known to be 
in their possession. The second guiding principle is that in the process 
of asking for information, one should never reveal more about a client 
than the person or agency already knew when they were first contacted. 

Introductory letters to former clients 

One approach for eliciting past clients' participation in follow-up 
involves sending a letter to those who are selected in the sample. As 
discussed in Chapter IV, the ethical considerations discussed in the 
federal regulations of confidentiality recommend two major precautions.^ 
First, no mention of alcohol or other drug treatment should be made in the 
initial letter to clients. Second, the study should be conducted under a 
neutral study name. 

Some suggestions for study names might be to use (with authorizatioi.) 
the title of an umbrella agency such as a health department, social service 
organization or research group . A study letter can also speak generally 
about a health survey and state that it is part of an effort to acquire 
information about health needs or treatment facilities. This is not to 
mislead clients, but to respect their privacy. 

The following guidelines should be considered when the initial letter 
to former clients is designed. 

1) The letter should not be on the program's letterhead. A 
neutral study name should be used (e,g,, Hennepin County 
Services Survey) . 



2) The return address and/or the return envelope should 
also state the neutral study name, using the program's 
address if that is where the interviewing will be scheduled. 

3) The letter must cover these basic points: 

...the purpose of the study (e.g., to study changes that 

occur in people's lives) 
...that clients have been specially selected for participation 
...that the client's participation is voluntary 
•..identification of the study's source of supporting funds 
...a guarantee of confidentiality 

...a phone number or address to contact, and directions for 
replying 

...notice of how further contacts will be made if the respondent 
is not heard from, and the length of time before the next 
contact . 

Obviously, the introductory letter should communicate clearly the 
intent of the study and the instructions to the clients. It is also 
helpful to emphasize the importance of clients' participation. Below is 
an example of an introductory letter which follows the above guidelines. 
It represents a conservative approach to confidentiality issues as suggested 
by NIDA publications on follow-up. 



106 



V-1 .') 



Study Title 

Date 



Dear 



You have been specially selected to be a participant in a research 

proiect we are conducting. The project is sponsored by ^ 

^ (fundinR source) 



under the direction of_ 



(name of director) 
The study concerns the changes that occur in a person's life and the 
effect they have on his/her well-being. We would also like to ask your 
opinion of services .provided by certain health programs in the community. 
The study findings will be used to plan and improve future programs. 

Your participation will involve giving us an interview for which we 
are going to pay you. Your participation is completely voluntary. An 
opportunity to inverview you will be very important to the research we are 
doing. 

We would appreciate it if you would call us as soon as you receive 
this letter. (If you are out of town, please call us collect.) This will 
allow us to explain more fully what the interview involves and to set up a 
convenient time to interview you. All of the information you give us during 
the interview will be strictly confidential and your name will not be used 
in any way. 

There is someone in our office every day between 8:30 A.M. and 4:30 P.M. 

Our phone number is . If we don't hear from you within two 

weeks, we will telephone you to explain further what our research involves 
and why we want to talk with you . 

Thank you very much for your cooperation. 

Sincerely, 

107 
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Instructions to office staff 

Wlien the Initial letter instructs clients to call to make an 
appointment or to ask any questions they may have, the office staff will 
need to be prepared to take these calls. The staff should have a copy 
of the letter on hand for reference, plus written instructions for making 
appointments and a list of questions respondents are likely to ask. Answers 
to these questions should be provided also. 

It may be helpful to install a special telephone line for the 
duration of the study so that calls related to the study can be received 
at one number. This will allow the staff to respond appropriately when 
the designated phone rings. 

Scripts prepared to guide staff when speaking to study subjects 
should be carefully designed. Ethical considerations suggest that no 
reference to chemical use or a study of chemical users be made because 
there is no guarantee that the person calling is really the research subject 
even though she/he may say so.^ 

Staff should be instructed to use the following guidelines: 

1) Answer the phone with the neutral study name. 

2) Do not mention the subject of chemical dependency treatment 
or chemical use. 

3) State that they are not at liberty to discuss all aspects 
of the study. 

As mentioned earlier, the purpose of these procedures is not to 
deceive the clients, but to protect them from unintentional disclosure. Not 
until the client presents him/herself at the study office or to an interviewer, 
should the complete nature of the study be revealed. 

^If a procedure is designed for assuring the caller's identity, these 
precautions would not be necessary. 
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Foilow-up arrangements and procedures will be iinL(|ue to ;m Individii.-il 



program and will, of course, require a script to comply with the study plan. 
However, the scripts listed below will serve as examples of how they can be 
designed to assist the staff. Note that the scripts contain no references 
to chemicals, chemical use or study of chemical users, thus respecting the 
privacy of people who might object to discussion of these issues on the phone 



How was I j)icked? 
Wliy was I picked? 

"Your name was selected from a list of many who could have been chosen by 
a scientific procedure called '"random sampling." This means that you will 
be used to represent a number of people, and instead of interviewing all 
of those people, we have chosen you. Your selection was a matter of chance 
and this is the way all of the people who are part of this study were 
chosen. If we fail to interview someone who has been selected, not only 
will that person hfi missing, but all of the people he/she represents will 
be missing from the study. That is why it is important to get your 
cooperation and to set up a convenient time to interview you." 

What questions will you ask? 
What do y .u want to know? 
What is the interview about? 

"In general, the kinds of things we will be asking about are the changes 
in your life and how your health has been in the last few years. If there 
are any questions that you do not want to answer, you don't have to, or 
if you should decide you would like to stop the interview for some reason, 
you will, of course, be free to do so. Naturally, we would like you *-o 
give us an interview that is as complete and accurate as possible." 

Why do you want to know these things? 

What are you going to do with the information? 

"The infomatlon we get from you will help us to understand the adjustments 
people make in their lives when things change for ::hem. Increasing what wa 
know about these things may make it possible to help improve program services. 



Questions likely to be asked by respondents 
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Where did you get my name /address? 
How did you get my name/address? 

"We have names of people who have used various services provided by 

agencies or programs in . However, we must protect 

(city or county) 

the confidentiality of our sources of information, namely the program 
and the people we interview. When you meet with the interviewer, he/she 
will be able to tell you more about how the participants were chosen." 



Who is doing this study? 

"The study is being conducted by a research group called , 

sponsored by . All of the information you 

give us will be used for research purposes only. Your name will not be 
attached to the interview, thus making it anonymous. All of the information 
is kept strictly confidential; yo'C privacy in this regard is protected 
by state and federal laws governing research studies." 

What is the study all about? 

"Basically, the study is concerned with the changes that occur in a person's 
life and the effect they have ou his/her well-being. It is not possible 
to explain the entire project over the phone, when you come into our office 
you will be given any further information you want." 

How long will the interview take? 
How much are you going to pay? 

"Some interviews run longer than others, however, it won't take longer 

than two hours. We want to pay you for your time. We'll pay you $ 

at the end of the interview." 



If phone interviews are to be conducted, an appointment could be 
made at the time a client responds to the letter. The interviewer would 
then be responsible for verifying that the client is the correct respondent 
and could then explain the study more fully. 



Initial phone contact 

If follow-up staff attempt to locate clients by phone without an 
introductory letter, there are some further precautions to be taken. 
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If cnM^ arc befn^; made to other persons to locaLc a lonner client, 
staff must also protect that (41ent's privacy. If asked, stafi could say 
they are working for a general agency, such as the State of Minnesota, 
and are interviewing people who have used various state services, but 
should not mention the program by name. The contact person can state that 
the person being sought has been drawn in a random sample and that the 
purpose of the call is to schedule an interview. However, this should not 
be offered unless it seems to be the only way to get the client to the 
phone or to find out where to reach him/her. A script explaining the study 
should also be prepared for the staff who attempt to contact follow-up 
subjects by phone. 

Informing clients of the study 

People may be reluctant to grant an interview for a variety of reasons: 
they may feel they are too busy; they may suspect the contact person is 
from some other state agency, such as corrections or revenue; or they may 
feel embarrassed. If the contact person senses this reluctance, he/she 
should be clear about tha nature of the study and emphasize the importance 
of the client's participation. 

It is necessary to provide an opportunity for staff to practice the 
procedures for contacting clients within a training session. At that time, 
all aspects of the contact can be tested to assure the staff's understanding 
and their ability to convey the information effectively. 

Q 

Problems after locating the client: Broken appointments " 

Broken appointments, after an interview has been scheduled, are a 
particularly common problem in follow-up studies. 
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The interviewer will need a strategy to deal with clients who do 
not show up or who are unavailable at the time of the scheduled interview. 
The following suggestions are made for re-contacting a client possibly to 
learn the reason for the broken appointment, but more importantly to 
schedule another interview time. 

Waiting time . If a client has not called to explain his/her lateness, 

or fails to show up for the interview within 30 minutes of the scheduled 

time, the interviewer should attempt to re-contact the client by phone. 

Re-contact conversation . When the call is made, the interviewer should 

identify him/herself by name and ask to speak to the client. If the client 

is not there, a message should be left that will 

(interviewer's name) 

call back. If the person answering the phone knows about the scheduled 
appointment, he/she may volunteer information as to where the subject is 
or when he/she will return home. If the information indicates that the 
client "forgot" or "couldn't make it," then the interviewer should respond 
in a positive way, stating that the subject should not worry about missing 
the appointment ("We'll just set up another appointment at another time.") 
Making another appointment . If the subject is present when the interviewer 
calls, the conversation should be pleasant and assertive, such as, "This is 
fYom • My calendar indicates that we have 



name study naire 

an appointment today for an interview. I thought I'd call to see if you 
were delayed or on your way, or to see if we could make an appointment for 
another time. Now, would tomorrow morning or Friday afternoon be better 
times for you, or do you think you still have time to come down today?" 
Arranging for transportation . If transportation seems to be a problem, 
another tactic that can be used is to center the conversation around ideas to 
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help the client. For example: was concerned that you might be 

having a problem getting a ride here, so I thought I'd call to see if you 
could take a cab or if someone from my office could come out to get you. 
If you take a cab, we will pay the fare both ways. Just have the driver 
stop in front of the building and someone will come to pay the fore." 
A friendly and understanding approach . It is also a good idea to suggest 
to subjects that probably something unforeseen had come up at the last 
minute to cause him/her to miss an appointment. The emphasis in this 
approach to **no-shows" is to accept whatever excuse the subject may have 
for not showing up, not questioning whether a subject really intended to 
come in or not. But if a person misses more than two appointments, a 
different approach will probably be more effective — one that displays more 
confrontation and less client accommodation. 

Changing the interview site . The purpose of an understanding approach 
is to allow the interview to be completed eventually. However, it may be 
necessary to change the site of the interview in order to secure completion. 
It may mean going to a client's home, picking a client up after work and 
conducting the interview then, or meeting the client at a restaurant and 
conducting it there. 

An appeal to reason . In the event that a client indicates that he/she is 
having second thoughts about participating in the study, a different approach 
should be used. Instead of trying to secure an immediate appointment, the 
interviewer should talk about the importance of the study and clients' 
participation, perhaps suggesting that the client think about it for a few 
days before the interviewer calls back to make an appointment. Generally, 
this appeal will keep the lines of communication open and in some cases will 
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result in a completed interview. In other cases, however, there may be 
nothing that will persuade a client to participate in the study. 

Pretesting the study procedures 

One of the most important procedures to be followed is the pilot 
study, or field test, of the follow-up scheduling and interviewing plan. 
A pilot study is a small scale study, on perhaps 10 to 20 ex-clients, which 
is done after the interviewers and other stp.ff have been trained and 
provided with documented procedures. 

The field test should incorporate all of the procedures outlined for 
the main study. It should involve only clients who will not be in the 
sample of the main study, but as much like them as possible. To ensure this, 
it should be done on those clients within the population from which the 
sample was drawn. 

The pilot study gives an opportunity to pretest the interview schedule 
and to revise the follow-up interview or questionnaire, when questions prove 
to be unclear or when categories of answers are inadequate. It very often 
will indicate the need to reduce the number of questions in order to keep 
the total interview time within limits the respondents will accept. It is 
also necessary to assure the interviewers ^ understanding of the instructions 
and intent of the questions. Each interviewer must record the same type of 
response consistently, in order to make the collected data reliable. 

Additional benefits include the identification of staff who are particularly 
good at certain functions, of those who may need further help and the develop- 
ment of some early cost estimates. 
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CHAPTER VI. DATA COLLECTION: COORDINATION, TRAINING AND SUPERVISION"^ 

by 

Judic J. Stelncr 
Rainbow Research, Inc.. 
Minneapolis, Minnesota 

The implementation of data collection is one of the most expensive 
and complicated tasks of a follow-up study because it involves the 
coordination of work for a number of people. Consequently, it is 
important for the effort to be managed with an eye tox^7ard avoiding waste 
while at the same time achieving quality in the data. 

One major reason these goals may be difficult to achieve is that 
the data collection process requires integrating the services of people 
who will locate and interview clients with those who designed the study 
and those who will eventually analyze the data. In most studies researchers 
and interviewers will work independently, without the benefit of each 
other's feedback guidance while the data is actually being collected. 
Since the follow-up coordinator will usually have the responsibility of 
conveying the study's intent, design and procedures to interviewers and 
other follow-up staff, a substantial amount of preparation will be 
necessary to provide comprehensive orientation for all of the people 
involved. 

This chapter outlines a number of factors to consider in order to 
control the quality of the data that is collected. It covers the following: 

1) Planning for interviewer recruitment; 

2) Selecting int rviewers: qualifications; 

3) Interviewer training: planning and content; 
4; Supervision and monitoring of data collection. 

•^This chapter of "Evaluation Tools and Procedures" was prepared under 
Contract #29055-29056 with the State of Minnesota, Department of Public Welfare. 
David S. Walonick, Steven E. Mayer, and Sue Weber assisted as editors of 
this chapter. 
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PLANNING FOR DATA COLLECT lON^ 

The project coordinator should have the method and arrangements 
for data collection clearly specified hefore he/she begins to recruit 
interviewers. Job descriptions for interviewers should include detailed 
responsibilities and a plan for how the work will be performed within 
the program's normal staff operations. 

Once the number of interviewers needed is determined, office 
space for scheduling and conducting interviews must be secured. It is 
important that the office space is private, neutral, and available 
exclusively for this function for the duration of the data collection. 
If more thaM one interviewer is used, it may be possible to schedule their 
work alternately within the same office space rather than obtaining a 
separate office for each interviewer. 

Some further consideration should be given to documenting scheduling 
and training materials in advance of selecting interviewers so that 
specific inquiries about the job can be answered. At a minimum, a job 
description, pay rate, time requirements, beginning and ending dates for 
the project, and any other necessities of the job such as a car, availability 
during evening hours or for travel, and any basic educational requirements 
should be determined before applicants are sought. 

A helpful procedure for assessing applicants during recruitment 
interviews is to ask them to interview someone using a simple q^iestionoaire 
with clear instructions. If they perform the task well and meet other 
criteria discussed in the following section, they probably are good people 
to select for the job. 
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J-Chapter IV discusses options for acquiring personnel to locate and interview 
former clients. It outlines advantages and disadvantages of three approaches: 
1) using existing staff of the program, 2) hiring additional personnel from 
outside the program, and 3) contracting for interviewers with a survey/ 
research organization. ^ . 
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SELECTION OF INTERVIEWERS 

Educational Background 

There are a number of characteristics that can be observed and 
Identified in candidates that will indicate their potential as effective 
interviewers. 

Generally, personality traits will indicate more about a person's 
qualifications for interviewing than past experience or educational level. 
Nevertheless, interviewers will need to be sophisticated in terms of 
working closely with people and be capable of administering the type of 
sensitive questionnaire that is often used in follow-up studies of 
chemical dependency programs. One way to give some assurance that 
interviewers will have the necessary ability is to require that they have 
had at least some college education as well as some experience working 
with people individually. 

Personal Qualities 

One essential qualification for interviewers is that they have a 
genuine curiosity about peoplvi and their experiences. This trait should 
be accompanied by good listening skills and sensitivity to life-problem areas. 

A second important characteristic is acceptance' of other people 
without judgment. Candidates should be able to ask questions in a supportive 
way that will encourage clients to be open and honest in their responses. 
People who are shy will not make good interviewers, nor will those who 
are aggressive, overbearing, or hostile. 

Third, candidates should be naturally compulsive and accurate record- 
keepers, and possess legible handwriting. Complete and accurate recording 
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is particularly important for follow-up staff who will be responsible 
for locating clients and providing summary data that document the 
attrition from the sample. 

Interviewing Experience 

The interviewing techniques used in follow-up are quite different 
from other interviewing skills, such as those used in counseling or 
personnel interviewing. Applicants with skills in interviewing for other 
purposes will still need special training to learn appropriate follow-up 
procedures and techniques. 

Matching Interviewers to Clients 

It does not seem to be important to match the age and sex of inter- 
viewers with the clients. Many studies have used both men and women as 
interviewers for predominantly male programs, and the quality of the data 
was not effected when interviewers of the opposite sex were used- 

However, studies that will deal with racial/ethnic groups exclusively 
will benefit from the use of interviewers from the same racial/ethnic 
background as the respondents.^ In some communities, an interviewer will 
have less difficulty getting information from respondents if he/she 
is not perceived as an outsider- Interviewers should also be able to 
give assistance in translation when English is not the first language of 
the respondents- 

If the sample has known proportions of various racial/ethnic groups, 
interviewers might be hired in the same proportions in an effort to 
match them with the respondents. 

^NIDA Publication - Conducting Follow-Up Research on Drug Treatment Programs. 
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Chemical Abuse/Dependency Background 

Many researchers have used interviewers previously trained as 
counselors in drug abuse/chemical dependency, expecting that they would 
be more successful in obtaining accurate information from clients. This 
is not substantiated by several large studies of chemical dependency 
programs that were done with interviewers from outside the chemical 
dependency profession, * In fact, it has been shown that there are 
advantages to using people somewhat removed from chemical dependency 
treatment, in order to enhance the objectivity of the study, ^ 

Since neutrality is one of the more important considerations for 
selecting interviewers, a candidate should first be able to avoid judgment 
and refrain from over-identifying with the respondents. Interviewers 
who have been through a chemical dependency program or who are chemically 
dependent themselves are more likely to identify closely with clients 
and therefore create a higher risk of bias in the collected data. 

On the other hand, a person who has never used any mood-altering 
chemicals would probably not make a neutral interviewer either. Obviously, 
finding life-long non-users to do interviewing would be difficult in our 
culture. The point is made only to illustrate extremes in interviewers' 
''identifying" capabilities. 

It is equally important for an interviewer to be comfortable and 
convey interest in a respondent. This behavior will increase the comfort 
of the respondent and do more to elicit accurate information than would 
an interviewer's chemical dependency knowledge. 

While questionnaire design will determine a great deal about the 
analyzability of the answers, the interviewers' technique and attitude 
will often be the determining factor of clients' willingness to respond 
honestly and appropriately to the questions, 

1 — 119 

^Ibid. 



Nevertheless, if a client chooses deliberately to distort the 
truth with his/her answers, it may be difficult to detect or control 
even for the most sophisticated interviewer, Tortunateiy, this type 
dishonesty can be minimized to a great extent by a non-threatening 
interviewer and a neutral environment • 



120 



INTERVIEWER TRAINING 



The training program for interviewers should focus on the objectives 
of the study. Basically, the goals are: 1) to collect accurate data, 

2) to collect data from as many respondents in the sample as possible, and 

3) to collect data within the limits of the project budget. 

To collect data that are accurate, interviewers will need to spend 
ample time developing their skills in administering the questionnaire 
through role-play practices. Certain aspects of interviewing technique can 
be learned through demonstrations, discussion and reading materials, but 
they cannot be developed as skills without sufficient practice. 

There are training manuals and video tapes available from several 
large research groups that teach interviewing techniques (listed in Appendix 
A of this chapter) . These training materials may be rented or purchased 
inexpensively. However, if adequate funds are available, programs may 
decide to use a survey research consi Itant to help develop and/or conduct 
the training. 

Planning a Training Program 

Some programs may wish to undertake their own training program for 
the people who will be used as follow-up staff. The fundamentals of planning 
training sessions for follow-up are summarized here as a guide for those 
programs. 

Length of training and teaching methods . An adequate training program can 
usually be done in three to four days. In order to minimize fatigue and 
maximize learning, each daily session should not run longer han seven hours. 

'^Some materials in the section are adapted from ^'Interviewing Respondents," 
by Celia Romans, Chapter 7, Conducting Followup Research on Drug Treatment 
Programs , of the National Institute on Drug Abuse Treatment Program 
Monograph Series, Number 2, pages 85^to 97. 
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Tr;jInc*eH will maintain a high level of Interest If the sessions ctmsist 
of varied teaching methods and techniques • A combination of reading 
materials, didactic instruction, group discussion and role-play exercise 
has been found to be an effective program for preparing follow-up staff • 
Appendix B contains a sample training program agenda ♦ 
Interviewing Techniques ♦ There are several important principles of 
interviewing. First, an interviewer should relate to a respondent in a 
pleasant manner, but also remain neutral and businesslike. In order to 
keep his/her manner from becoming social, an interviewer may have to 
assert directives to continue on with the interview, should a client begin 
talking about subjects not related to the interview. However, the inter- 
ventions made by the interviewer must be tactful. For instance, the 
interviewer might say, "It is interesting to hear about your operation. 
Perhaps you can answer the question which concerns your length of stay in 
the hospital, if I re-read it." 

With regard to other distractions, such as inappropriate hospitality, 
the interviewer must try to avoid anything that creates difficulty or 
interferes with the process of completing the interview in as short a time 
as possible. 

The second principle for interviewing is that the questionnaire must 
be followed and the questions read exactly as they are written. This is 
imperative for ensuring that each respondent is answering the same question. 
Emphasis must be made that even a slight change in a question can change 
its meaning entirely. For example, "When did you go to the treatment center? 
is considerably different from "When did you last go to the treatment center? 
Obviously, valid comparison cannot be made without knowing respondents 
have answered identical questions. 
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The third principle of interviewing involves a technique called 
probing. Probing in a way to get additional information, a clc»arcr 
answer, or a fuller description of an opinion, without influencing a 
respondent's answers. Probing techniques are discussed further in the 
sample interviewer manual. Appendix C of this chapter. 

Interviewers need to Icnow the purpose of the questions, flow of the 
questionnaire (including "skip" patterns, where the next question depends 
on the answer to a previous question), the probes that apply to certain 
types of questions and also problems that may arise during the interview, 
some of which are discussed in the appendices of this chapter. Interviewers 
should be provided with a manual developed for the specific study for use 
during the training and for reference while data is being collected. 
Confidentiality and privacy . All aspects of confidentiality and privacy 
should be clearly presented during the training. Documentation of procedures 
to protect client privacy should be included in the interviewer's manual 
to assist them when they contact and interview respondents. To help secure 
the privacy of data, it should be required that all project information be 
placed in locked files when it is not being used. In addition, the 
coordinator should closely oversee the procedures for removing identifying 
information and storage of completed forms. 

Another aspect of confidentiality is the risk of interviewers' verbal 
disclosure of information outside of the project. It should be emphasized 
that their role as interviewers is one of committed confidentiality and any 
breach of this rule should be considered grounds for dismissal. To help 
ensure this, interviewers should be instructed not to discuss interviews 
with anyone outside the project, even if names are withheld. However, it 



is helpful if interviewers are provided an opportunity to discuss their 
experiences and share interesting interviews during periodic conferences 
with the project coordinator/supervisor. 

Attitudinal issues . It is rare, indeed, to encounter a person who does 
not have an opinion about drug problems, and who these problems effect 
as a result of his/her own experience. Some steps toward attitude awareness 
and increased interviewer objectivity should be taken during the training 
by furnishing pharmacology and chemical abuse information that will help 
dispel myths and stereotypes of people with drug-related problems. Beyond 
providing education, this approach can assist interviewers in identifying 
their biases (beliefs) and will increase their awareness of how they might 
unconsciously attempt to influence client responses, whether verbally or 
non-verbally . 

Presentation of issues surrounding drug related problems can be 
dealt with in a number of ways. Videotapes, films and reading materials 
are available from many sources. The Drug Information Service Center at 
the University of Minnesota has an exceptional library for this purpose. 

It is essential that the drug-related information present arguments 
from both conservative and liberal perspectives, so that the trainees 
can place themselves somewhere on the continuum of attitudes. A list of 
topics for discussion and titles of available recorded information is listed 
in Appendix A. 

The goal of the attitudinal session should be to assist interviewers 
in identifying their positions on the attitudinal continuum and to recognize 
why they placed themselves there. It should be made clear to trainees that 
they have a right to their own opinions, but that those opinions should not 
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cvlcJonL ;it ;iny time during their conlnt-L with cliunlH. This me;ins 
that It is inappropriate for an interviewer to counsel or advise clients, 
even if clients ask for the interviewer's opinion. 

The only suggestion that an interviewer might make is by way of a 
referral to an agency or counselor. However, a referral should not be 
given unless the client asks for help. A referral should not take place 
during the interview but should be put off tactfully until all of the 
questions have been answered. Instructions for giving referral information 
to clients who request help should be covered in the training along with 
familiarizing trainees with local resources that can handle a variety of 
problems. 

Practice interviews . As noted earlier, a large amount of training time 
must be spent familiarizing trainess with the questionnaire and practicing 
interviews. Initially, the questionnaire should be reviewed question by 
question, clarifying the intent and instructions for each item. 

It is a good rule to require each trainee to do several role-play 
interviews, varying the respondents (played by another trainee) in terms 
of chemical use pattern, life-style and post-program attitude. As the 
training progresses, role plays should be seasoned with some unexpected 
difficulty, possibly portraying some identified problem areas for the trainee. 
It is essential that each role-play is discussed by the group after it is 
completed, to give trainees feedback about their delivery and probing skills. 
Critique and suggestions should be made by demonstrating alternative behaviors 
that are more appropriate. 

The practice inteirviews may indicate the need to rearrange the 
questionnaire items to improve the flow of questions, to reword questions 
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or to include additional response categories. Therefore, to help 
clarify and improve the instrument, it is wise to request the presence 
of the research designer at these sessions. 

The training is also where checks on reliability of interviewer 
recording takes place. As part of the role-play exercise, all of the 
trainees should record the answers of a respondent, as though they 
were conducting the interview that is being observed. When the interview 
is finished, the group then compares their recording of each response. 
When differences appear, the trainer must indicate the correct way of 
recording the response. This exercise will help assure consistency in 
interviewers' recording patterns. 

Scheduling procedures . Contacting and locating clients, arranging a 
place and time for interviews, and ways for dealing with difficult situations 
should be clarified through review and discussion of written instructions. 
A procedure for "checking out" the trainees' comprehension of the 
instructions is to have each trainee role-play a situation from the initial 
contact through editing the completed questionnaire. This exercise 
will often raise questions that will indicate where the instructions are 
unclear. This will give the trainer an opportunity to reiterate instructions 
and/or modify them as needed. In addition, preparation of the study 
records, such as retrieving information from client files, maintaining 
control files and accounting for the sample must be communicated thoroughly 
and checked out through observation by the trainer. 

Miscellaneous and special instructions . Some studies will require more 
detailed and specific instruction than others, depending on the procedures 
needed to accommodate the scope of the project. Some of these special 
options for follow-up are discussed below. 
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1) I>D, cards or letters of introdtict: Ion . Wlicn interviewers I Lrst 
meet with respondents, they may be asked to show their identity indicating 
their affiliation with the project. Identification cards may be printed 
showing the "neutral" study name along with the phone number of the 
study office. This way, if there still is some doubt, the respondent can 
call the study office to verify the interviewer's name. Letters of 
introduction from the study director are also effective when used for 
this purpose. 

2) Introduction by interviewer . Instruction to the interviewer 
should include direct and to the point responses for answering respondents' 
questions regarding the purpose of the study and what good it will do them. 
Emphasis should be made that the purpose of the study is to find out 

about both good and bad experiences in the program and that the study will play 
an important role in the design of future services.^ By conveying the 
importance of the study and asking for negative as well as positive 
coi-ments, the respondent will likely feel more comfortable in being 
honest with his/her answers. 

3) Informed consent and legal forms . Interviewers may be assisted 

by documentation that can be shown to respondents explaining confidentiality, 
clients' rights, and privacy, and also laws which govern the conduct of 
research, such as those that may state conditions for the release of 
clients' names to research groups. 



^Chapter IV discusses scripts for follow-up staff in more detail. 



SUPERVISING MD MONITORING DATA COLLECTION 

IntervJewers work very indopendont ly , which mo.Miis tlicy r;in 
occasionally lose touch with the goals of the study and with details they 
learned in training unless they have regular contact with the project 
coordinator. Careful supervision is essential in the first few weeks 
of the study and will require interviewers to report their progress 
frequently. After this period, the coordinator should check interviewers' 
work at regular intervals. 

Validation of Interviews and Editing 

Validation of a completed interview consists of the coordinator 
contacting a respondent again to verify that an interview actually took 
place and includes asking a few questions from the interview to see that 
the information is accurate and complete. This procedure may not be 
appropriate for some studies, particularly when a coordinator can observe 
how much time the interviewer is spending with a respondent, if interviews 
are coi.ducted in one main office. However, it is advisable to validate 
some interviews, selected at random, when they are conducted in the field. 

The procedure for validation should be developed with the following 
considerations : 

— Preparing the respondent by explaining the procedure and 
that someone may call to validate the interview. 

— Keeping the manner of validation consistent with the promise 
of confidentiality. 

— Things to verify might be: 
— Date of interview 
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— Length of interview (allowing approximately 30% 
discrepancy) 

— Birth date to verify respondent's identity 

— Several questions from the interview (limited to those 
the client is likely to answer the same way and knows the 
answer to) 

— Validators should not know the original answers to the questions 
beforehand. 

— Validations are usually conducted by phone. 

When discrepancies are found, the interviewer should be relieved from 
working for a couple of days, so that more validation of his/her work can 
take place. 

If there is consistent evidence of disagreement with validation, 
then all the interviewer's work should be checked and steps should be 
taken to remedy the inaccuracies. 

After the interview has been checked by the interviewer, it is also 
a good procedure to have a member of the follow-up staff check the 
completed questionnaires for skipped questions and errors in recording. 
Errors should be discussed with interviewers on a regular basis for the 
purpose of correction. 
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APPENDIX A 



SOURCES FOR INTERVIEWER TRAINING MANUALS 



"University of Michigan Manual" 
Publications Office 
Institute for Social Research 
University of Michigan 
Ann Arbor, Michigan A8106 



National Opinion Research Center (NORC) 
6030 South Ellis Avenue 
Chicago, Illinois 60637 



Videotape — "Interviewing Techniques" 
Mathematica 
P.O. Box 2392 

Princeton, New Jersey 085A0 



Suggested topics for discussion for attitudinal assessment: 

— Psychology of intoxication^ 

— Theories of chemical dependency; disease concept, behavioral, 
predisposition and hereditary theories (all are discussed in the 
Rand Report, Alcoholism and Treatment , Amour et al. Chapter 2, 
"Perspectives on Alcoholism and Treatment" 

— Illicit vs. licit drugs (including prescriptions) 

— Pharmacology, effect of drugs/alcohol 

— Polydrug use 

— Alternative and traditional treatment 

— Implications for prevention and early intervention of drug problems 

— Definition of drug-related problems, e.g., negative consequences 

— Labeling people chemically dependent 

— AA philosophy 



Workbook available: Psychology of Intoxication , by Cullen et al. 
(Drug Information Service Center, University of Minnesota). 
Film and videotape: "Attitudes Towards Intoxication. A Model for 
Understanding Drug Taking Behavior" by Dr. John Brantner, Department 
of Clinical Psychology, U of M (also available from DISC). 
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LEARNING OBJECTIVES FOR INTERVIEWER TRAINlNc: 
TO ASSIST PARTICIPANTS IN THE FOLLOWING: 

1. Developing an understanding of persons experiencing chemical use 
problems . 

2. Recognizing their oto attitude regarding drugs and drug users. 

3. Understanding the prevention and development of chemical use problems. 

4. Discovering resources available for more extensive diagnosis, 
client referral, and additional information. 

5. Understanding intended impact of service delivery under study and 
purposes of follow-up. 

6. Understanding conduct of follow-up with regard to confidentiality issues, 
1. Application of recording and coding procedures for the questionnaire • 

8, Developing interviewing techniques. 

9. Developing schedules for interview and data collection. 

SAMPLE TRAINING PROGRAM AGENDA 
Day 1. 

9:00 a.m. Welcome, overview of the training program 

9:15 Review of the study; purposes and goals 

9.30 The program; philosophy, clientele, and objectives 

10:00 Attitudinal issues (presentation by a qualified person); 

social, prescription, and illicit drugs, theories of chemical 
dependency 

10:30 Break 

10:45 Videotape "Psychology of Intoxication" 

Group discussion 

11:45 Confidentiality, study design and procedures for protecting 

client privacy 

12:30 Lunch 

1:30 Drugs and their effects (presented by a qualified person); 

pharmacology, and poly-drug use 
Group discussion 
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2:15 Treatment modalities — Traditional and Alternative 

Review of printed materials and discussion 

3:00 Break 

3:15 Review of interviewing techniques 

Day 2 

9:00 a.m. Discussion of previous day, reading materials, and manual 

9:30 Content of questionnaire; clarification (question by question review 

10:30 Break 

10:45 Tasks before the interview: Locating and contacting clients; 

arranging time and place for the interview; dealing with 
difficult situations 

11:30 Role-play interview (trainees play both roles) 

12:30 p.m. Lunch 

1:30 Discussion of role-play and interviewing techniques 

2:00 Revision of questionnaire (if necessary) 

3:00 Break 

3:15 Survey Administration: Record-keeping; editing questionnaires; 

checking for completeness 

Day 3 

9:00 a.m. Role-play interview (from initial contact to editing questionnaire) 
10:30 Break 

11:00 Discussion, comparison of recording for reliability 

12:30 p.m. Lunch 

1:30 Scheduling the field test (pilot study) 

2:00 Role-play interview 

3:00 Break 

3:15 Discussion of interviewer attitudes and expected problems 

(e.g., safety, burn-out, bias, hostility from respondents, 
feelings while interviewing) 
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Day 4 

9:00 a.m. Role-play interview (comparison of recordings) 

10:30 Discussion of reliability and accuracy 

11:00 Break 

11:15 Interview critique from trainer to individual trainees 

12:00 p.m. Lunch 

1:00 Review of procedures — Confidentiality and forms 
(clarification if necessary) 



3:00 Field-test assignments* 



*A field test is conducted soon after training. Trainees return as a 
group to discuss the experience, review completed forms, and make 
necessary revisions in the procedures. Chapter V discusses field testing 
procedures, also known as a pilot study. 
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APPIiNDIX C 



SAMPLE INTERVIEWER MANUAL^ 



Role of the Interviewer 

The role of the interviewer is to obtain information. When the 
interview is completed and the participant has requested assistance or 
services regarding problems covered in the interview, the interviewer may 
then suggest or refer the client to an appropriate person or service. 
Even at the end of the interview, the interviewer's role will not be one 
of a counselor, but one of an agent or resource person. 

While this type of interview is not designed to penetrate deeply 
into the personality of the respondents or to change them in any way, 
it is possible that their awareness of themselves will be heightened. It 
is important that the interviewer be sensitive to the client's problem 
areas and aware of local resources that may be appropriate to the client's 
needs. 

The role of the interviewer is crucial to this project. Without 
valid, reliable data collected by the interviewers, the results of the 
study will be worthless. To this end, it is important that the interviewers 
not be seen as agents of the program, but as someone who is neutral and can 
be trusted with honest feedback. 



Confidentiality and Privacy 

Interviewers are required to protect the confidentiality and privacy 
of all participants involved in the study. Each participant and her/his 
family is to be treated with respect and dignity, as is all information 
they provide in carrying out this study. There are several general rules 
to follow to prevent breaches in confidentiality and privacy: 

1. No personally identifying information will be on any 
questionnaire or form submitted to the data collection office. 

2. Use all precaution necessary to prevent loss or theft of 
completed forms. At home, keep forms in a safe place. 

3. Do not allow anyone other than project staff to read 
completed forms . 

4. Do not discuss participants with anyone other than project staff. 

5. At your workplace, keep all personally identifying information 
and completed forms locked up when not in use. 



This manual contains study specific information that may not be 
applicable to other follow-up projects. 
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Initial Contact 

Your first contact with potential participants will be either by 
phone or in person. This is a critical time and the interviewer must 
keep several important aspects in mind, since the potential participant's 
first impression may influence his/her perception of you and the study. 

The success of being granted an interview (and successfully conducting 
the interview) depends considerably on the interviewer's ability to 
create a friendly, non-threatening atmosphere of mutual trust and 
confidence. Because the conditions of each contact are varied, there is 
no single "right" way to establish rapport. Interviewers must be sensitive 
to the uniqueness of each situation, and respond accordingly. 

There is, nevertheless, necessary information to share and certain 
principles to follow that are likely to bring about rapport. The following 
information should be given in the first conversation: 

1. Identify yourself and who you are working for. 

2. Describe the purpose of the study. The purpose of the study 
should be stated broadly. A general statement on the study with 
emphasis on the importance of their thoughts and opinions 

will usually suffice. 

3. Explain how the information will be used. Assure the participant 
that the information given will be seen only by the study staff 
and that in reporting findings no names or identifying 
information will be used. Also, make clear that the study is 
being conducted independently from the program staff operations. 

4. Participation is voluntary. The . participant should feel free 
to withdraw his/her permission at any time; however, emphasize 
that they can contribute important information we need, that this 
is an important study and they would be helping us. 

5. Explain what is expected of the participant. Make it clear that 
the appointment will be made at their convenience. The time of 
the interview should be no more than one hour; it is better to 
over-estimate the time allotment than underestimate it. 



Scheduling by Phone 

When someone other than the participant answers the phone, caution 
and discretion must be exercised. Confidentiality must be the primary 
consideration. The main concern is that no one should know, without the 
express permission of the client, that the client was in a chemical 
dependency program. 
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If the person answering the phone is definitely known to be an 
immediate family member (spouse, parent, etc.) living at the same 
residence, it might be acceptable to identify yourself as someone 
conducting a survey. Do not, however, volunteer this information but 
offer it if it appears to be the only way to get the participant to come 
to the phone. 

If the interviewer is calling other persons when attempting to 
locate a participant, do not give any information that reveals that 
the client has ever been in a chemical dependency program. The interviewer 
may say she/he is working for the State of Minnesota, interviewing people 
who have used various state services, but do not mention the program by 
name. State only that the person you are trying to locate has been 
drawn in a sample and that you would like to talk with this person to 
ask if he/she would be willing to be interviewed. 



Other Considerations 

People may be reluctant to grant an interview for a variety of 
reasons: they feel they are too busy, they may believe the interviewer 
is from some other State agency such as Corrections or Revenue, they may 
feel embarrassed. If interviewers sense this reluctance, they should be 
clear about the nature of the study and emphasize the importance of 
former clients' participation. 

An interviewer should not attempt to interview a participant whom 
they know (relative, friend, acquaintance). This situation could be 
uncomfortable and bias the interview data. If the interviewer has prior 
knowledge of the participant or her/his family, contact the coordinator to 
have this interview reassigned. 

If, in the course of the interview, you have reason to be concerned 
for your safety, do not prolong contact to complete the forms. Excuse 
yourself as tactfully as possible and report this to the coordinator. 
One of two alternatives will be decided: (1) to assign two interviewers 
as a team to see the participant; (2) to remove this person from the 
interview sample. 



Interviewing Techniques 

1. Create a neutral atmosphere. 

The most important criterion in an interview is the validity of the 
participant's responses. Many people will respond with what they 
think the interviewer wants to hear or what they think the "right" 
answer should be rather than giving a true answer. This makes it 
extremely important to respond neutrally by showing no signs of 
approval or disapproval, whatever is said. 
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1. Bor language should be relaxed. 

2. Facial expression should be pleasant. 

3. Verbal responses that are appropriate are: "I i^ee," 

"Uh huh," "OK, I've eot that." Avoid expressions such as 
"good," "fine," "oh no," and other slang terms of approval 
or disapproval. 

Display interest. 

Without becoming too friendly, do not appear cold or distant — 
possibly a ''detached - interest . " Maintain eye contact when the person 
is responding. 

Behavior of the client. 

It is permissible fcr the interviewer to indicate disapproval of 
the participant if he/she is uncooperative (or approval if very 
cooperative). To discourage uncooperative behavior, use a straight- 
forward remark such as: "The interview will take longer if you will 
not (stay in thC; room) (pay attention)." 

Communicate that the interview is important. 

Participants are more likely to cooperate if they believe the 
interview is serious and purposeful. This may be conveyed by: 

1. How thr study is explained initially. 

2. The interviewer's conduct and attitude (verbal and non-verbal). 

Make it clear that the respondent's answers are important. 

Respondents may feel that they are different from the "norm," or 
that their extreme opinions will ruin the study. The interviewer 
should convey that no matter what the situation, all answers are 
important, thus increasing cooperation. 

Request a quiet place for the interview. 

This will help emphasize the importance of the study. If a phone 
interview is granted, request that the participant eliminate distractions 
before beginning the interview. 

Conduct the interview privately. 

It is definitely preferable to conduct the intar'^iew in private. 
The presence of even one other person may influence the respondents* 
answers, and therefore bias them. 

Difficulty in responding to questions. 



Be alert to problems a client may have in responding to the questions, 
if: 
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1. English is not their first language. 

2. A learning disability, visual or auditory handicap is present. 

3. Their educational level Is low. 

A. They are confused duo to senility, brain (lama|^,e or medlratlons. 

The interviewer should then pace the interview and gear his/her 
vocabulary to facilitate the understanding of the client. In severe 
cases, it may be necessary to forego the interview completely. 

9. Clarification to respondent. 

When a question is not understood, misunderstood or misinterpreted it 
should be repeated, not paraphrased. Pose the question more slowly 
or begin by saying, "Let me say it more clearly," or "I see," and 
repeat the question. 

NOTE : A respondent may say "I don't know" to a question he/she 
does not understand. Though sometimes genuine, this response should 
not be accepted right away at face value. Repeating the question 
or probing, may be successful in bringing out a more valuable answer; 
however, do not force an answer. 

10. Hesitation or refusal to answer questions. 

Questions which respondents hesitate or refuse to answer immediately 
should be handled tactfully in order to keep rapport. The interviewer 
should use the introductions supplied on the questionnaire in a casual 
manner, without concern about t-.n sensitivity of the question. In 
other words, there is no need to prepare the client fcr a particular 
question as it would only serve to make them feel defensive or uncomfortable. 

NOTE : Part of maintaining a relaxed and neutral atmosphere is the 
ability to ask the questions in a natural, matter-of-fact way; not 
appearing to read too closely to the format. 71 the client is still 
hesitant, you may remind his/her of the anonymity of his/her answers. 

11. Identifying inadequate responses. 

The interviewer must understand the objectives of the questions in 
order to identify unclear, incomplete or untrue answers given by the 
respondent. When there is a need to pursue a suspect answer the 
following rules should apply: 

1. Probes must not suggest reponses. 

2. Acceptable probes in cases of contradiction or unclear 
responses are: 

a. "Earlier you said..." 

b. "Could you say more about the fact that..." (repeating 
the response) 

NOTE: Maintaining rapport may necessitate reminding the respondent 
that there are no right or wrong answers. An irrelevant answer will 
also require a probe to elicit a more appropriate response. 
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12. Probing with preceded questions. 

With precoded questions, probing is used to focus the respondent's 
attention to the question or direct him/her to the existing 
categories. In all cases, when the answer does not correspond to a 
category, it should be recorded verbatim before it is probed. 

Answers "between categories" 

For example: QUESTION: How would you rate your health—excellent, 
very good, good, fair, or poor? 

ANSWER: It's OK. 

Wlien this situation arises, the interviewer should repeat the 
question (indicating to the respondent that they must choose one of 
them) by 1) repeating all categories to avoid narrowing the respondent's 
choice, 2) prefacing the repetition with some remark. For example, 

the interviewer might say, "Well then, would you say " or 

"Which comes closest" or "In general, ." (Never just rattle off the 

question again.) 

If the respondent should say, "It depends," then a remark such as 

"In general. " is appropriate. On the other hand, if the respondent 

uses his/her own words instead of a category, then "Wliich comes closest" 
is best. 

13. Probing with open-ended questions. 

Open-ended questions are used when the full range of possible answers 

is not known by the researcher. Consequently, it is particularly 

important to get complete information from the respondent. Two kinds 

of probing are required: 1) for clarity and 2) for additional information. 

The first task of the interviewer is to evaluate the answer to be 
sure it is understood in writing, not only by the interviewer but also 
by others who will analyze the data. If it is not clear, the interviewer 
must decide to probe for a better understanding. It may mean asking 
for a more specific response or for an explanation of a term. 

Probes for clarity tell the respondent what the interviewer needs to 

know, such as: "What do you mean and repeatiing 

the response, or "I want to be sure I know what you mean here." Vcter 
the answer is clarified, give some indication that it is understood, 
such as "That's the kind of specific information I'm looking for." 

Remember: It is important to be sure to probe tc the point of the 
question, and not ask for superfluous information, or reasons for 
reasons . 
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When probing for additional information, r"ipcatinK the subsfanro 

of the question with a request for further information is appropriate. 

For example: "What else do you like about '-^r 

"What other reasons did you have • 

The probe "What else?" should not be used, because it tends to 
sound too mechanical and the respondent may get sidetracked. "Is 
there anything else?" should be avoided because it is a leading probe 
that can be easily answered by a NO. 

Never probe until the respondent says, "That's all." The point is to 
probe until the respondent really has nothing else to say. 

lA. Closure of the Interview. 

1. Review the questionnaire quickly to see if there are any skipped 
(unanswered) questions before ending the interview. With the 
exception of those questions that are not applicable, all questions 
must be answered. 

2. Extend thanks to the respondents for their participation and 
appreciation of the time they ;.',ave to the study. 

3. Commimts on conduct of the interview mcy be recorded after 
cont&ct with the client has ended. Note problems or circumstances 
that may have influenced the interview on the last page of the 
questionnaire. 

A. Editing of the questionnaire should be done in pen (if OLlginal 
recording is in pencil), as soon after the interview as possible. 
Check each question to be sure that the answers are correctly 
checked or circled, written answers are legible and abbreviations 
are spelled out. Never erase an answer, either during or after an 
interview. Never fill in any information based on what you think 
a respondent might have said (the errors made using this latter 
"judgment" are greater than incorrect recordings during the interview). 



Scheduling Participants 

The purpose of the original letter is not only to inform the client 
of the study but to also assist in scheduling an interview. It is quite 
likely, however, that many people will not return the scheduling portion of 
the letter within a reasonable time period. Therefore, it will be necessary 
to make another attempt to reach them. If phone numbers are available, 
the client should be called to see if he/she received the letter. The 
interviewer should try to get a verbal agreement to participate in the 
study as well as set up an appointment f<^r an interview. If at all possible, 
try to avoid tentative arrangements. A coiranitment to a specific time and 
place is more apt to produce an interview, than a promise to call you back. 



Uo 



VI-27 



For those clients who do not have listed phone numbers, a reminder 
letter should be sent approximately two weeks after the first letter has 
been mailed. The interviewers will be responsible for designing a short 
note, requesting an immediate response. The urgency should relate to 
scheduling within a given time period, due to the relatively short time 
in which we have to complete the study. 

When the scheduling portion of the letter is returned and it 
indicates a positive response to participate, the interviewer will respond 
as quickly as possible. They will naturally have the first priority of 
the interviewer. 

The client roster will be used to indicate the progress of returned 
letters, undelivered letters and scheduling of interviews. It may be 
necessary for an additional notebook or card file to record contacts, 
lead information and time spent on each client. The interviewers are 
encouraged to design a comfortable system for themselves that will provide 
the needed status information for each client. 

As clients are contacted and responses received through the mail, 
status information will be recorded in the roster and the summary sheet. 
The categories will include the following: 

Summary sheet : 

1. Number of clients contacted — contacted in person and 
scheduled or mailed a questionnaire. 

2. Interviewed — completed interviews (mail or in person). 

3. Refused — said no to interviewer or in the mail. 

4. Ineligible — client in treatment, prison, etc. 

5. Deceased or incapacitated — long-term hospitalization, etc. 

6. Unlocatable — after all leads have been tried. 

7. Open or scheduled — interviews not completed but scheduled 
or clientii not yet reached. 

The clients from whom nothing is heard (meaning no forwarding address 
or other lead) will be shown as open until the end of the s^ady. We will 
assume the letter reached them unless it is returned. Reminder :3 may be 
sent periodically to try to obtain a response. 

The number of completed interviews should be broken down on the 
summary sheet between phone, in person and mail interview, during a week. 
A monthly total of all categories, including the number still open at 
the end of the month, will be submitted to the evaluation office. 



NOTE : The number of clients contacted means only tliosG who ma^ be 
categorized, i.e., refused, interviewed, etc. ICveryone else will be 
considered "open or scheduled." At the end of the month the number 
contacted plus the number open should equal the total of the sample. 

Average Time Per Interview 

Each interviewer will spend a certain amount of time to collect 
data, contact clients, travel (when necessary), and interview. A record 
of time spent on each client who has completed the interview will be 
entered on the roster (log). 



CHAPTER VII. THE PREPARATION OF DATA FOR ANALYSIS 

by 

David Walonick 
Rainbow Research, Inc. 
Minneapolis, Minnesota 

One of the most difficult tasks of the researcher is preparation 
and analysis of the data. This phase of the follow-up study requires 
a full understanding of the study design and the methods used to collect 
the data. To a large degree, this task will depend on how well each 
of the previous research steps have been performed. The results of any 
statistical analysis can only be as accurate as the data collection 
techniques. If interview questions are ambiguous or the original purposes 
of the study are not clearly stated, the task of analysis will be difficult, 
if not impossible. In a well planned follow-up study, the methods of 
analysis will have been designed at the onset of the project, not after 
the data have been gathered. 

PLANNING THE ANALYSIS 

Planning the analysis goes hand-in-hand with planning the interview 
questionnaire. Both are dependent upon the defined purposes of the study. 
Questionnaire items are designed to answer specific questions pos. d by the 
study. They are also designed to enable a specific type of analysis. In 
other words, performing a statistical analysis v/ill depend on the series 
of steps that have been incorporated into the questionnaire. 

One of the first decisions to make in planning an analysis is 
whether or not to use a computer. In small studies with relatively few 

This chapter of "Evaluation Tools and Procedures" was prepared under 

Contract //29O55-29056 with the State of Minnesota, Department of Public Welfare. 

Steven E. Mayer, Judie Steiner and Sue Weber assisted as editors of 

this chapter. 
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cases and questions, the use of a computer may be unnecessary; however, 

some analyses are quite complex and would require extensive time to do 

the necessary arithmetic by hand. In large studies with hundreds of 

cases and variables, it may not be possible to do an analysis without a 

computer. The cost of computer time has steadily decreased in recent 

years and will probably continue to do so. A typical two-page questionnaire 

can usually be analyzed for under ten dollars of computer tine. (Of 

course, the exact cost will depend upon the complexity and extent of 

the analysis, as well as the number of cases and variables involved.) 

The major costs of computer analysis are in preparing the data for analysis 

(coding and keypunching) and hiring a computer programmer. 

There are several packaged computer programs that have been designed 
particularly for social science research. The most commonly used is 
SPSS (Statistical Package for the Social Sciences).* It has been developed 
to accommodate a large variety of research methods and will provide the 
user with a fast, comprehensive analysis. Unfortunately, the successful 
use of SPSS requires either an experienced programmer, or considerable 
time to learn how to use the program. Still, it can reduce the amount of 
time needed for analysis from months to days. 

PREPARING OPEN ENDED QUESTIONS FOR ANALYSIS 
Pi.eparing data for analysis can be either a time consuming task or 
a relatively easy task, depending on how well the analysis was planned 
before the data were collected. If a questionnaire was ''thrown together," 
the time needed to prepare the data will be considerable. On the ot ir 

*Norman H. Nie, et. al.. Statistical Package for the Social Sciences, 
McGraw-Hill, Inc., 1970. 

o 
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handy a well planned questionnaire will reduce this time to a minimum. 

Hood analyses are both descriptive and inferential. They describe 
the sample and allow inferences to be made about the population. That 
is, they should allow a summary description of the sample in terms of 
typical responses and variability, and they should allow use of standard 
statistical ''tests of significance" to yield estimates of what is likely 
in the larger population.* 

Open-ended questions present especially difficult problems in analysis. 
Since each respondent used his or her own words to answer the questions, 
it is unlikely that any two respondents answered the same question in 
exactly the same way. A question asked to 100 different people may elicit 
100 unique answers. While a review of the verbatim responses may be 
valuable in itself, it is more meaningful for summary purposes to combine 
the responses on the basis of common response themes. The task of 
developing appropriate response categories for open-ended questions is 
difficult and time consuming. In addition, it allows the researcher's 
own bias to be introduced into the results. 

For example, ten people were asked the question: "When you have an 
emotional problem, whom do you usually confide in?" The following ten 
responses were received: 1) my girlfriend, 2) my mom, 3) my wife, 
4) my sister, 5) my brother, 6) my doctor, 7) my Aunt Sally, 8) my lawyer, 
9) nobody, 10) my husband. 

*This chapter cannot explore the variety of procedures used in the actual 
analysis of data. The reader is referred to the following textbooks: 

Guilford, J. P. Fundamental statistics in psychology and education. 
NY: McGraw-Hill, 1965. 

Haber, A. & Runyon, R.P. General Statistics Reading, Mass: 
Addl son-Wesley, 1969. 
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Response categories could be constructed in several different 
ways. One way ia: 

1 = Confide in member of the immediate family 

2 = Confide in someone else 

3 = Do not confide in anyone 

This set of response categories might have been constructed by a 
program specifically interested in learning whether family participation 
in the program would create more trust in the client (as opposed to a 
non-family member). All the responses received could be easily coded 
into one of the thi categories. The new categories could then be sub- 
jected to a statistical analysis. 

It is imperative that all possible responses are codable into the 
available response categories. One way to assure this is to include an 
''other" category or "none of the above." A good techni^iue for creating 
response categories to open-ended questions is to select randomly 
approximately 20 completed questionnaires. After creating response 
categories from these, another 20 questionnaires should be chosen for 
coding into the categories to test their applicability. If they fit 
well, the categories will probably be sufficient. 

Creating categories for verbatim responses is basically a data 
reduction technique. In other words, very precise data (the actual 
verbatim responses) is being reduced into less precise data (the coded 
response categories) . 
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NOMINAL, ORDINAL, INTERVAL & RATIO DATA 

There are four different kinds of data. For illustrative purposes, 
the following questionnaire was developed to show the differences- 
(1-3) 1. Follow-up ID number 

(4) 2. Sex 

1 = Male 

2 = Female 

(5) 3. How often have you drunk alcohol in the last week? 

1 = Not at all 

2 = On one or two occasions 

3 = On three or four occasions 

4 = On five or more occasions 

(6) 4. Overall, how would you rate our program? 

1 = Excellent 

2 = Good 

3 = Fair 

4 = Poor 

5 = Worse than poor 

(7) 5. How many years of school have you completed since 

high school? 

(8-9) 6. Age (in years) 

This questionnaire is in a format conducive to a computer analysis. 
Key punchers will be able to work directly from the questionnaire itself. 
The numbers in parentheses represent the columns on the computer cards 
where the responses will be punched. There are several very important 
differences between these questions in terms of the kind of statistical 
summaries that can be done with each of them. 

The first question is the client's follow-up ID number. The ID 
number is simply a substitute for the client's name. Its only use is to 
identify who completed the questionnaire. The number itself has no 
arithmetic value (i.e., Client 002 is not more of a client than Client 001) 
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The second question is similar. The respondent's sex is coded 
into a response category. The numbers 1 and 2 merely stand for a name 
(male or female), and have no arithmetic value in themselves. That is, 
a code two is not twice as much as a code one. No arithmetic operation 
on the numbers can be done except counting the frequency of each code. 
For example, a mean average sex cannot be computed. (i.e.. The average 
sex of the sample is not 3/5 male and 2/5 female.) This type of data 
Is called nominal. It contains valuable information, and proportions, 
percentages and rates can be calculated. Nominal data is often used to 
describe a sample. By counting all the cases that have a 1 in column 
four (males) , and counting all the cases that have a 2 in column four 
(females) , the sex composition of the sample will be known and can be 
•( "..''nd as an actual count, as a percentage of males or females, or as 
the ratio of males to females. The modal average (most frequent response) 
is also sometimes reported for nominal data. 

The third item on the questionnaire ("How often have you drunk alcohol 
in the last week?") is somewhat different than the first and second items. 
On this question, the response numbers are ordered.* A person with a code 
2 drank alcohol more often than a person with a code 1, and a person with a 
code 4 drank more often than a person with a code 3. The numbers however, 
do not indicate how much more. This type of coding (scaling) is referred 
to as an ordinal scale. The responses are ordered in a systematic way, 
but the numbers do not represent equal quantitative units. Ordinal scale 
numbers can be tricky. At first glance, it appears that an average frequency 
of drinking episodes could be computed by adding all the response numbers 

*A good textbook on measurement, as opposed to statistics, is by 
Nonally, J. "Psychometric Theory." NY: McGraw-Hill, 1967. 

lis 
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and dividing by the number of cases. Closer examination shows that an 
average calculated with ordinal scale numbers will be quite misleading. 
Since the coded response numbers only show the relative orHr^r of responses, 
they cannot be added together (i.e., a person with a response coded as a 
4 did not necessarily drink twice as often as a person with a response 
coded as a 2) . The numbers are ordered (ordinal) , but the intervals 
between the numbers are not necessarily equal. However, mode and median 
averages can be calculated for ordinal data. 

Attitude scales are also usually ordinal data. Question four 
asked respondents to rate the program on a five point scale. Respondents' 
answers will not tell how much difference there is between two people 
with different responses. It would be inappropriate to compute a mean 
average rating for the question. 

While the ordinal scale is very helpful to the researcher, it is 
lacking in the ability to quantify differences between responses. Some 
kinds of data can be scaled so that intervals between the response categories 
are known. Generally, if there is a unit of measurement for any piece of 
data, an interval scale can be established. 

Item five on the questionnaire (How many years of school have you 
completed since high school?) is interval scaled. The unit of measurement 
is "grade level." It is considered interval scaled because the intervals 
between grade levels are equivalent. A response of zero would indicate 
that the respondent has not completed a year of school beyond high school. 
This "zero-point" is not an absolute number. That is, a zero on this 
scale does not indicate the "absence of education." 
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Item six on the questionnaire asks the respondents' age. The 
unit of measurement is "years." Unlike the interval scale of the previous 
question, the "zero-point" for this question is absolute. It can be 
said that response of 40 is twice as old as a response of 20 and a mean 
average age could be computed for the sample. This kind of scale is 
referred to as a ratio scale. Like the interval scale, each numeric 
interval represents one unit of measurement. Ratio scales also have the 
property of an absolute "zero-point," 

When possible, ratio scaled questions are preferrable in research 
design, since they offer the most versatility in the kinds of analyses 
that may be performed. Often, ratio scaled data will be recoded to an 
ordinal scale for the purpose of analyzing and reporting the information. 
It's usually much easier to make sense out of the data when presented 
in an ordinal scale. For example, it may be more desirable to recode the 
respondents' ages to the ordinal scale: l=under 21, 2=21 or older. It 
is easier to understand a summary based on these response categories 
than a list of all the ages and how many people are each age, 

Recoding any data into new response categories is a procedure 
that should be approached with caution. Since it is basically a data 
reduction technique, it is possible that the recoded responses will 
present a distorted picture of the true responses. Recoding data is 
another procedure in which researcher's bias can be introduced. 
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PREPARING DATA FOR NON-COMPUTER ANALYSIS 

When a computer will not be used in the analysis, the depth of the 
analysis -^'111 probably be limited by time. The traditional method of 
organizing data for a non-computer analysis is to prepare a tabulation 
sheet on which the responses to each item for each person can be 
recorded. A tabulation sheet for the five item questionnaire is 
illustrated in Figure 1. Each case requires one row on the tabulation 
sheet and each variable (questionnaire item) requires one column. These 
columns are not the same as computer card columns. 

Using a tabulation sheet consists of counting the appropriate 
cells for each case. Calculations using this method are surpri?, ingly 
fast when there are few items or cases. On the other hand, it is somewhat 
limited in its applicability to crosstabs or subgroup analysis. 



FIGURE 1 



ID // 


Sex 


How often 
drunk 


j Program 
rating 


Educ. past 
High school 


Age 


001 


2 


1 


1 


0 


17 


002 


2 


3 


2 


1 


29 


003 


1 


4 


5 


3 


31 


004 


2 


2 


2 


4 


58 


005 


1 


3 


3 


4 


43 


006 


1 


2 


4 


0 


38 


007 


1 


4 


5 


0 


27 















VII-10 



PREPARING DATA FOR A COMPUTER ANALYSIS 



Preparing data for a computer analysis is a relatively easy 
task and can be performed by someone who has had no previous experience 
w^th computers. In a well planned study, the questionnaire itself will 
have been designed so the data can be keypunched directly from the 
questionnaire onto computer cards. If the questionnaire was not designed 
to facilitate keypunching, a lot of time will be needed to transfer 
information from the questionnaires to a computer coding form. This is 
usually a tedious process. The implication is clear: A little extra 
effort in questionnaire design man a big savings in preparing the 

data for analysis. 

Most modern data prO)ces^. ing services now have the equipment to 
"key to disk" rather thair. keypunch computer ca^rds directly. The "disk" 
is a mass storage device. If a mistake is made in keypunching (on computer 
cards), it is necessary to repunch a new carc3 completely. If a mistake 
is made when "keying to disk," it ig an easy matter to correct it. When 
all the data has been entered and stored on disk, computer cards can then 
be punched quickly and accurately. 

It is always important to have data "punched and verified." During 
the verification process, errors will be detected and corrected. After 
spending a large sum of money on the collection of the data, it would be 
foolish to jeopardize the reliability of the information by not having 
it checked. Good research design dictates data verification. Data card 
punching and verifying costs between 5 and 10 cents per computer card. 

When data is being prepared for computer processing, it must be 
structured in a form that a computer can understand. The computer card 
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is an aid for helping convert written questionnaire information into 
computer Information. Computer cardn rontain 80 columns, each capable 
of holding one character of information. This character may be alpha (A-Z) 
or numeric. 

Each item on the questionnaire will require at least one column 
on the computer card, and some questionnaire items will require more 
than one column. For example, if the questionnaire asked the respondents' 
age, two columns on the computer card would be reserved for this item. 
This would accommodate all possible ages to 99 years. The number of 
columns required to hold a variable is called the "field length." The 
required field length for age is two. Another example where two columns 
would be necessary is when there are ten or more response codes for a 
particular item. On the sample questionnaire, the numbers in parentheses 
are the coliunn numbers where the responses will be placed on the computer 
cards. 

Each case is begun on a new card. If the questionnaire requires 
more than 80 columns, it is continued on another card. It is customary 
(but not necessary) that the beginning columns of each card contain the 
case ID number and the card number for that case. This would allo^ the 
cards to be sorted if the card deck in dropped or becomes misordered. 

Figure Two shows the first two cases of a questionnaire that requires 
two and a half cards per case (200 columns). Columns 1 through 3 on each 
card are the ID number and column 4 is the card number. The second case 
is started on a new card even though there are unused columns remaining 
on the third card. 
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FIGURE 2 
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0013 
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There are several questionnaire formats conducive to keypunching. 
Most have two characteristics: 1) the column numbers for each question 
are printed on the questionnaires themselves, and 2) the completed 
questionnaires will have the responses in a column so the keypuncher will 
be able to look down the page in a straight line, rather than zigzagging 
back and forth across the page. This will help reduce the keypunching 
error rate. For an extensive discussion on questionnaire design, the 
reader is referred to Questionnaires; Design and Use by Douglas Be. Jie 
and John Anderson (Scarecrow Press, Inc., 1974). 

If a questionnaire was not designed to be directly keypunched, the 
information must be transferred from the questionnaire to an intermediate 
coding form, which in turn will be used by the keypunchers. This step 
is time consuiiiing and highly susceptible to error. The entire 
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reliability of the study is dependent on the accurate transfer of 
information from the questionnaires to the computer coding forms. 

A computer coding form resembles the format of the computer card. 
There are 80 columns on each line of the form. (Each line will be punched 
on one computer card.) As mentioned earlier, the major problem with 
transferring information to coding forms in accuracy. Coders will need 
to be hired and trained to perform this task. The primary requisites 
for this job are attention to detail and endurance. It is imperative 
that response codes are written in the proper columns on the coding 
form. Skipping a column will lead to disastrous results. It is 
therefore extremely important to spotcheck coders' work. If a coder 
misunderstood the instructions, spotchecking his or her results will 
detect this and prevent future headaches. It is also a good idea to 
reserve a column for the coder's ID number. If this is done, it will be 
possible to identify the coder for each questionnaire, and thus allow 
an easy check and correction procedure for consistent errors by a 
particular coder. 
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MISSING DATA 

Missing data is of.^en a problem in survey-type studies. For 
example, in a written questionnaire, 100 former clients were asked: 
"Did our program help you to make any good changes in your life?" The 
response categories are: l=yes, 2=not sure, 3=no. In the sample, 
45 respondents answered yes, 15 clients said they weren't sure, 10 said 
no, and 30 left this question blank. How to interpret these results 
is a difficult problem. The 30 who left this question blank could sway 
the results any direction. 

The first thing that should be done when large amounts of missing 
data are encountered is to try to obtain the data. While contacting each 
respondent who left this question blank may seem like an extreme measure, 
it may be the only way to make sense out of the data. Of course, the 
amount of effort that goes into getting the missing data will depend upon 
the importance of that particular question to the study. 

Often, it will not be possible (or practical) to contact the 
respondents for the excluded data. This will especially be true when a 
study is constrained by time or budget. If this is the case, confidence 
in the results will be diminished. 

Generally, there are two ways of reporting results that contain 
missing information. The first method is to create a new response 
category for those who didn't answer the question. (in the example, 
4=no response.) Statistical calculations are based on the total sample 
N, and missing data is treated as if it were another response category. 
No assumptions about why the data is missing need to be made. Using 
this technique, the results might be presented as: 
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45% Yes 

15% Not sure 

10% No 

30% No response 

In interpreting these results, the researcher would still look 
for the reason so many responses are missing, but would not exclude thos. 
people from the analysis. This technique is particularly suited to 
nominal and ordinal data. If the sample is not significantly different 
from the population from which it was drawn, a similar amount of missing 
data would be expected from other samples of the population. 

The second method of reporting missing data can be used with any 
type of scaling. The data for the question is summarized as if the 
sample size were equal to the number of people who answered the question. 
(In our example, the sample size for this question would be 70.) The 
percent of people who answered yes to this question would be calculated 
as 45/70 X 100 = 64% (rather than 45/100 x 100 = 45%). The results of 
this question might be reported as: 
64% Yes 
21% Not sure 
14% No 

Response rate = 70% 

When the data is interval or ratio scaled, this technique is often 
carried one step further. The mean average of the completed sample is 
calculated and all missing data is given this value. The mein is then re- 
calculated using the total sample. It is important, when using these 
techniques, to report either the response rate or the actual number of 
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missing cases. This will give other people aa idea of how much 
confidence they can place in the data. Ri»H()onHe rates uf Iorh than 
90% are generally considered low, and decisions based on low response 
rates should be made with caution. 
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BRANCHING 

Branching in a questionnaire means that respondents are asked 
different questions depending on how they answered previous questions. 
The simplest form of branching is illustrated in the following example: 

1. Have you drunk any alcohol in the last month? 

l=Yes 
2=No 

2. If yes, how many times? 

l=On one or two occasions 
2=0n three or four occasions 
3=0n five or more occasions 

Question #2 will only be answered by those who answered "yes" 

to question //I. If there is no missing data, then the number of respondents 

to question //2 will equal the number of people who said yes to question //I. 

Branching can present problems in an analysis if there is missing 

data. This is especially true when data is missing from the question where 

the branching occurs. In the sample questionnaire, if question //I has 

missing data, then there will be no way to determine how many respondents 

should have answered question //2, Because of this potential problem, 

branching should be avoided when possible • It is more desirable to 

phrase questions so they can be asked to all respondents. 

In conclusion, good research design requires planning the analysis 

simultaneously with the questionnaire, so that most analysis problems can 

be prevented. 



CHAPTER VIII: REPORTING AND' USING 'THE RESULTS OF A CLIENT FOLLOW-UP STUDY-"- 

by 

Steven E. Mayer 
Rainbow Research, Inc. 
Minneapolis, Minnesota 

The previous chapters considered the planning and execution of 
a client follow-up study, from formulating the study objectives to 
preparation of the data for analysis. This last chapter considers the 
final steps: writing an evaluation report, and using the results. 

First, a format for presenting the results is shovm, as borrowed 
from traditional "science reporting." The uses of narrative, tabular 
presentation, and graphic displays are discussed. 

Next, several "uses of follow-up" as previously cited in Chapter 1 
of this Handbook are re-introduced to show what can be done with the 
evaluation report. Several issues from the evaluation subspecialty called 
"knowledge transfer" or "utilization of results" are presented to show 
sources of organizational resistance to change, along with a few 
recommendations to the evaluator. 



This chapter of Evaluation Tools and Procedures was produced under 
contract r/29055-29056 to the Minnesota Department of Public Welfare. 
This chapter was edited by Sue Weber, Judie Steiner, and David Walonick, 
of Rainbow Research, Inc. 
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WRITING AN EVALUATION REPORT 



The results of the follow-up study should be written up, along 
with the procedures used to conduct the study. Written presentations of 
t'y^ research projects typically contain several standard sections: 
Introduction, Method, Results, Discussion, and Summary. A number of 
issues should be addressed within each section. 

Introduction 

The Introduction to the report should describe the context of the 
study. It should describe the program under study, including the program 
clientele, the specific services that define the program, and the program's 
treatment philosophy. The more detail that is provided in describing the 
program, the more easily the reader can understand the type of program 
which the results describe. Without this detail, the reader cannot know 
to what types of programs or clients the results might be applicable. 

The Introduction should also describe why the study was undertaken, 
and under those sponsorship. The specific purposes of the study should be 
listed. The section could discuss similar studies or studies previously 
done with the program, and other historical information that might help 
to give further context to the study. 



The Method section should document the procedures used to conduct 
the study. The ideal is to include enough detail to allow other investi- 
gators to conduct an identical study of another program, or at the same 
program in the future. At a minimum, it should be detailed enough to allow 
readers to understand how the data were collected and analyzed. 



Method 
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The Method section also should describe how the sample of clients 
to be followed-up was chosen. It should explain what types of clients 
were eligible for participation and how clients were chosen. 

Since not everyone on the sample list will actually provide data, 
an accounting of sample attrition should be given. This would include 
a breakdown of reasons why people in the original sample were not among 
the final sample. Reasons could include "coult not locate," "refused to 
participate," "institutionalized," "lived too far away," "deceased," 
"unreliable interview data," etc. 

The method for contacting clients and the information they were 
given prior to their participation should be discussed, along with steps 
taken to assure the client's privacy and the conf identaility of the data, 
A description of the data collection instrument, whether a written 
questionnaire or an interview form, should be given. If the instrument 
was a standard one, or used in other studies, its name and availability 
should be given. If the instrument was "homemade," a detailed description 
of its development is needed, and a copy of the instrument itself should 
be included as an Appendix, At the least the questions should he included 
in the test, perhaps as part of the Results section. Individuals or 
groups of people who participated in constructing the instruments, or other 
parts of the study's design or implementation, should be mentioned too. 

If data were collected by interviewers, the Method section should 
describe the interviewers' relation to the program, and any special skills 
or qualifications they brought to the job, A description of the training 
or orientation they received is needed. The "set and setting" for the 
interview transaction should be described. If data were collected by 



mailed questionnaires procedures used for reminding clients to return 
(luestlonnalroB, and the success of thesci procedures shouJd also hv 
discussed. 

Results 

The Results section should present analyses and summaries of the 
collected data in a way that is responsive to the study objectives. One 
of the most difficult tasks of report preparation is settling on the 
"right" format and level of detail in which to present results. Ultimately 
the decision is based on one's understanding of the needs of the different 
audiences of the report. 

One could present a summary of the results of each question on 
the instrument. This could take the form of a frequency distribution, 
showing the number and percentage of the sample that answered the question 
in each of the response categories, and could be broken out further by 
giving the frequency distributions for different population subgroups. 
For example, the responses to a question could be shown separately for 
males and females, or for people discharged one year ago and five years ago. 

As further examples.^ c le could present all the data for one subgroup 
first, and then another. Another option is to present all the data 
according to component of the program. Still another is to present the data 
according to the original study objectives. Obviously, different combinations 
of the above arrangements can be designed. (See Figure 1 for an example.) 

A well-constructed table is an economical way to present data, and 
can allow the reader to perceive relationships that might be missed if the 
results were reported in strictly narrative form. 



A good table supplements the text. Similarly, each table should be 
integrated into the text. The text should state the highlights of a 
table, rather than all the detail. Kach table should bo intelligiblo 
without reference to the text. It should have a clear title, each of the 
columns and rows should be clearly labelled, the units or numbers reported 
in the table should be identified, the sample size should be given, and 
qualifications or exceptions should be presented as footnotes to the table. 
(See Figure 2.) 

Each table should be numbered, and referred to by its number in 
the text. If the same table format is to be used repeatedly, it is usually 
wise to mock up a prototype so that headings, labels, and terminology 
can be standardized to facilitate comparison among tables and to avoid 
confusion. 

Data presented graphically, rather than in tabular or narrative 
form can greatly aid understanding of the data. A visual expression of 
data adds meaning or emphasis to a strictly numerical or verbal presentation.^ 

The most common and simplest forms of graphic presentations are 
pie charts, bar charts, and line charts. The pie chart (See Figure 3) is 
a circle whose area is divided into segments. It is especially useful for 
showing the relative size of the segments, and for comparing the segments to 
the total. It is effective for displaying how a total group is made up of 
smaller groups. Pie charts with too many pieces can be coxif using, they are 
often difficult to label clearly, and comparison of several pie charts can 
be clumsy and imprecise. They are effective visually, but should be used 
sparingly. 



^An excellent handbook on the visual expression of data is by 
Mary Ellen Spear, Practical Charting Techniques , published by McGraw-Hill, 196 
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QUESTION: Do clients say the ref-^ral was helpful? 

ANSWER: At follow-up, the question was asked, "Did it do any good to go 

to the place or places suggested by the counselor f'-om the program?" 
Results are shown in Table 5, in the right panel, for both internal 
and external referrals. 

" For those who acted on the refprral... 
...52X said it was "a big help" 
...39% said it was "some heln' 

9X said it was "no help" 

3S said it did "more harm than good" 

• - ^orty-eight percent of clients with problems related to their 
own use found the referral "a big help." Fifty -five percent 
of those with problems related to someone else's use found 
the referral "a big help." Forty-eight percent of those with 
problems unrelated to chemical use found the referral "a biq 
help. " 

" Of those who felt pushed or 'orced to use the Cnvernor's Bill 
prografi, 27% felt that the program's reconmendation for further 
assistance wts "a big help." Fifty-nine percent of those 
rot pushed or forced felt the referral was "a biq help." 

— People who finished the recommended program, or H>?re still 
receiving services at the time of follow-up, were more likely 
to find the program "a big help" (625) than were people who 
left before finishing (27T of these said it was "a biq help"). 
Fifty-three percent of the people who ^aid the program was 
"no help" left the service before completing the program. 



Figure 1. An Example for Presenting Results. From 

Client Impact Study of Chemical Dependency 
Prevention and Early Intervention Programs 
fo r Special Populations. Volume I: Results 
for the Total Sample . Minneapolis: Rainbow 
Research^ Inc. , 1979 
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Table 8 



Ti'-fle explains 
^reup names. 



Mean Nu mber of Co rrect R esponse^. on Verbal Testa 

. — >— >^ 

by Children With and Without Pretrainlng 

Grade 3 Grade 6 



In ihble. 




20 (18) 

20 (19) 

20 (19) 

20 (20) 



280 
240 

281 
232 



319 
263 

317 
262 



Nolo. Maximum score = 320. 



(o«^n«le Numbers in parentheses Indicate the number of 

{X column 

e«pi4/ns children who completed all tests. 
u%c of 



Figure 3,. Example of a well-constructed table- 

From P ublication Manual of the American 
Psych ological Association ^ 2nd Edition. 
Wai ; ■x"- 7ton, D.C., APA, 1974. 



A bar chart (See Figure 4) is useful for comparing different items 
at a given time. For example, one could display proportions of different 
populations of people who report a particular outcome or event, or the 
cost of different items or services at a given time. It effectively 
displays the relative magnitude of different classes, and is userul when 
there is no need to compare the different items to the whole (in which 
case a pie chart might be better). It is also possible to compare two 
groups, or sometimes three, using bars that are coded by different shadings. 

A line chart (see Figure 5) is also an effective way to express data. 
Typically, it is used to show how some quantity (cost, numbers of people, 
outcome rates) change over time. It is especially useful a) to show 
change over a long period of time (five or more units of time), b) when 
several series can be plotted simultaneously (like numbers of admissions 
and numbers of ^'successes*') , c) when the emphasis is on the rate of change 
more than the actual amounts. 

Discussion 

The Discussion section of the report should contain an assessment 
cr interpretation of the results, especially with respect to the study's 
objectives. It could contain a summary of how the study's objectives were 
met, or how the original questions .^rere answered. Departures from original 
expectations should be noted. Similarities and differences between results 
of the study and results of other studies could be noted. 

The Discussion section should also include the investigators' 
judgments of the validity and quality of the study's results. If the data 
are not as conclusive as originally expected, reasons should be offered. 
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Figure 4. Example of a bar chart. From Client Impact Study 

of Six Detoxification Centers ("Sub-Acute Receiving 
Centers") . Minneapolis: Rainbow Research, Inc., 
1979, 
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Figure 3. Example of a pie-chart. From Client Impact 
Study of Six Detoxification centers ("Sub- 
Acute Receiving Centers") > Minneapolis : 
Rainbow Research, Inc./ 1979 
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PAGE 2k 



RGUFEV ■ 

Crlm lUt«s* and CI««ranc« Rut*! for P«rt I lnd«x Crimes Ramsay 
County from 1973-1977 
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Figure 5. Example of a line graph. From 1977 Ramsey 

County Crime Analysis . St. Paul/Ramsey County 
Criminal Justice Coordinating Council, 1979. 
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Various sources of bias chat may have entered Into the sample ultimately 
interviewed, biases in interviewing or other phases of the data collection 
process, and biases in analyses should be examined. 

Difficulties encountered in conducting the study should be discussed, 
to serve as a guide to future investigators. Difficulties from the stand- 
point of project administration, or study design and execution should be 
discussed. 

The Discussion section is also the place to discuss any implications 
the study may have for changes in theory of chemical dependency, programming, 
or administration. It is usually difficult to decide how far from the 
actual data one can safely depart before entering the realm of speculation 
or personal attitudes. One should probably avoid polemics and vague 
theoretical wanderings, and instead be guided by the questions, "What has 
this study contributed? How has this study helped to resolve the original 
problem?" 

How free one is to recommend particular courses of action is 
contingent on several factors. One is whether it is expected by the 
study's sponsor. Another consideration is the degree to which recommendations 
are inspired directly by the data, or by one's own judgment. Still 
another rests on the technical quality of the study; the data may look 
compelling, but be seriously flawed, making the base for policy change 
recommendations somewhat unstable. 

The data may clearly indicate that a change is needed, but the 
exact nature of the change, or the mechanism for bringing about the change 
is probably not directly indicated by the data (unless it was deliberately 
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designed into the study). If recommendations for change are made, 
they flhould be supported by the presented data. The investigator should 
determine whether the report, or some other vehicle, is more appropriate 
for expressing recommendations for change. 

Other approaches 

There is nothing hard-and-fast about presenting an evaluation 
report in the sectioned format just discussed. It has worked well in 
scientific journals, and is probably workable for other audiences and 
settings as well. Certainly, the issues presented above do need to be 
conveyed somehow, regardless of format. 

Many style manuals have been written to guide editorial consistency 
and quality. These are often published by a particular profession's 
association, or endorsed for use by the major journals of that profession. 
If one contemplates publishing the results of an evaluation research 
project in a professional journal, it is advisable to consult back issues 
of the journal to ensure that the format used will be compatible with 
publication requirements. It is often helpful to consult style manuals 
for helpful pointers on clarity, regardless of one's intent to publish.-' 



3 

An example is Publication Manual of the American Psychological Association , 

2nd ed. Published by APA, 1974. Also helpful is Manual of Style , 

University of Chicago Press, 1969; Style Manual , U.S. Govt Printing Office, 1973, 
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USING THE RESULTS 

Chapter 1 presented several "Uses of Follow-up Studies." While 
these potential uses should have guided the planning and conduct of the 
study and the preparation of the report, now is the time to plan organi- 
zat: viial responses to the evaluation findings. It is most likely that 
the report will not have touched on, or made obvious, all the ways in 
which a program can be improved, so that it will be necessary to expand 
and promote ideas from the report. Again, the responsibility for creating 
and promoting an organizational direction should not be assumed, but 
should be clarified early in the project and asserted as it becomes 
appropriate. 

Follow-up as contributing to program evaluation 

Are the program's objectives being met? Administrators should 
assess how their view of objectives-attainment is enhanced or modified by 
the evaluation. A major contribution of follow-up data is that they 
provide the client's perspective one that is probably less familiar than 
other, more customary (such as staff's) perspectives. While an evaluation 
study may not attest to all program objectives, there should be some 
conscientious staff work around the question, "What have we learned about 
our program's objectives from this study?" 

Is there program impact of which the program was unaware? Unless 
the study design was extremely restrictive, there will probably be feedback 
from clients about topics that may be surprising. Some clients may have 
been effected in ways that were unanticipated by the program. The report 
should be scrutinized for impact the program did not know it had, and for 
new ways of describing impact, both good and bad. 
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What new evidence is there os the program's strong and weak points? 
It is quite possible that the feedback provided by the reporr. gives an 
unfamiliar picture of the relative strengths of components of the program, 
or the relative impact on different areas of a person's life. Even 
though there is a tendency to read an evaluation report for the way ±t 
confirms or disconfirms one's prior expectations, one should try to learn 
what new knowledge about the program has been added by the report's 
assessment of 3trong and weak points* 

Follow-up as contributing to program design 

Many areas for improvement will probably be revealed in an evaluation 
report. Some will be revealed by direct feedback from clients. For 
example, upon learning that less than one-third of the people in a follow- 
up sample find the lectures or films on chemical dependency helpful or 
informative, a change in selection of films or lectures is indicated. Or, 
learning that women often find it humiliating to be forced to wear 
pajamas in a "co-educational" setting should be incentive enough to prompt 
a change in that practice. 

Sometimes a shift in program emphasis is indicated by an analysis 
of the data. For example, it was discovered that in detox centers one of 
the few practices to be rated as highly helpful or supportive is the 
practice of including representatives of community groups or special 
population groups in the supportive care of the person in detox. This 
practice is presently uncommon, but should be considered for more regular 
inclusion In program practice. As another example, programs could very 
well learn that many of tMeir former clients lead happy and productive lives 
even though they do not accept much of the program's philosophy. This 
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could suggest that the program could enlarge its definitio i of treatment 
success, as transmitted from counselor to clients in the program. 

Follow-up as contributing to staff development 

A major purpose of follow-up data is that they will provide knowledge 
of results to the staff. This is a primary ing^.edient to learning; without 
knowledge of results, one cannot improve one's practice, v^ile counselors 
do not usually have the opportunity to get knowledge of results on a case- 
by-case basis (unless the counselor did the follow-up inter 'lew her /himself ) 
there is opportunity to learn from aggregate data. By seeing the variety 
of ways in which the program is perceived -ind received by program clients, 
counselors will have more realistic expectations of the program's impact 
on clients. 

An assessment of the program's strengths and weaknesses should lead 
to an assessment of training needs for program staff and administrators. 
Clients will often say directly what areas a counselor was weak in, or what 
areas the entire program ignored, or detail administrative complaints 
against the program, or comment on areas of insensitivity or incompetence. 
These should be converted into a plan for staff improvement. 

Follov-up as contributing to discussion on "effectiveness" 

A major product of a follow-up study is the way it can serve to 
provoke discussion on what is effective programming. Even though the report 
will not answer all questions about effectiveness, it will certainly have 
the effect of raising many questions which are better informed, given the 
new data. It could be revealed, for example, that standard notions of what 
is an "appropriate referra" i-'e missing some important ingredients that are 
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revealed In the data. Or it could bo {hni *V|unlfty sobriety" Ls a 
notion tfiat defies simple description. 

The situation is not unlike a good detective story, in which 
clues and evidence unfold over a period of time. The first set of clues 
may point broadly in one direction, providing "leads" to be pursued. 
Occasionally there will be false trails and embarrassing moments as 
seemingly good clues turn oi^t to be value-less, but with thoughtful 
pursuit the lines of evidence become more firmly drawn. Perhaps the test 
of a valuable study is whether the concept of program effectiveness after 
the study is finished is further developed than it was before the study 
began. 

Often, an evaluation report is produced and njthing happens. The 
uses for which the report was originally intondad may not be realized, 
and the report may sit on someone's desk interminably. What happened? 

At conferences of evaluation researchers, considerable time and 
energy is used for discussions of why evaluation data are not used more 
by decision-makers or change agents. What are some kn<-»vn factors that 
constrain the usefulness of evaluation reports?^ 

1. Organizations are remarkably resistant to change » 

The status quo is usually comfortable, and change typically 
requires new energy, resources, and cooperative soul- 
searching. Programs are typically imbedded in complex 
social systems, with histories and habits that make change 
an unsettling experience. 



^Some material for this section was taken from "The Process of Program 
Evaluation" by John Van Maanen. The Gransmanship Center News. 
Vol. 5, No. 1, 1979. 
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Programs have Interests other than achieving 
their stated objectives. Program survival and 
personal needs for job security may tend to make 
administrators and staff cautious, preferring not 
to "rock the boat." They may also become political 
in ways that serve the occupational/professional 
needs of the staff, rather than the bona fide needs 
of the client aad community. 

The changes recommended by an evaluation process 
may not be ideologically acceptable to the program. 
When recomme-iudations or findings threaten basic ideology 
or professional allegiances there is bound to be serious 
opposition. The classic example in the chemical dependency 
field is, of course, the denial by some factions of data 
supporting the legitimacy of a variety of treatment approaches 
and outcomes. 

No study will answer all questions, making it an easy 
target for criticisms of inadequacy. Sometimes, a response 
to study results is to a shrill insistence that there should 
be more data, and a devaluing of the data actually available. 
Of course, all data will have some methodological short- 
comings which can bring into question the credibility of 
the effort. 

There is often the expectation th?t an evaluation effort 
will present, or should present, a definitive statement on 
the program's effectiveness. In practice, of course. 
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evaluations typically provoke more questions than 
they answer. This should be turned into a virtue, so 
that evaluation can be recognized as a process that 
leads to progressively greater understanding of effective 
performance. 

Relationships between evaluator and program staff may 
turn from cooperative and mutually supportive to a state 
of wary coexistence and even hostility. 

The program itself may have changed substantially during 
the course of the study, rendering the outcomes of the "old** 
program obsolete. This problem os substantial changes in 
the program is most critical when the study is of the 
controlled-experiment variety, where treatment changes in 
mid-experiment make the results uninterpretable . 
The reasons that an evaluation was requested originally 
are varied, and not always entirely honorable. It has 
become a practice in some places to request an evaluation 
study, rather than appoint a committee, as a way of post- 
poning a decision, realizing that an evaluation can take 
even longer than committee work and is politically easier 
to ignore. 

Evaluation studies sometimes do miss the mark. The 
techniques of evaluation are still often crude, often used 
inappropriately, cr produce findings of questionable validity. 
Even if valid from a research standpoint, their usefulness 
for decision-making can be severely restricted. 
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There are a few things under the control of the evaluator that 
might he^p to ensure greater attention from the prog^-am to the evaluation 
effort. 

1, The early involvement of program administrators and 
staff in the evaluation effort is very important. No one likes 
to have an evaluation imposed on them, and involvement by those 
P<^ople affected by the evaluation should help to decreose resistance 
and Increase cooperation. 

2, The program could be kept upda^ted continually on progress, 
and on preliminary findings. 

3, The evaluator should be candid about the lima nations of 
the evaluation effort. Evaluators are still learning their craft, 
and this should be acknowledged by all parties. 

4, Evaluators should realize that substantial program change 
is not a likely result of the evaluation alone, and should adjust 
their own and others' expectations accordingly. Presentation of an 
evaluation report can be a uneful and even major part of a larger 
change process, but there is a gap I: ^t ween data and action. The gap 
can be filled by the principles of ''planned change" or "organizational 
development,'* professional specialty areas with a history and 
literature of their owr . ^ 

5, The Human Interaction Research Institute, published a study 
on factors influencing the success of NIMH-funded applied research, 
not necessarily program evaluation. By way of summary, they stated 

^See, fc^r example, Bennis, W. , Benne, K. , and Chein, R. (eds.), 
' ih^ Planning of gh ^^, 2nd ed. , (New York: Holt, Rinehart, and 
Winston, Inc., 1969) or Beckhard, R. Organizational Developiient; 
Strateg i es and Models (Reading, Mass: Addison-Wesley , 1969). 
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that the successful project was "characterized by high communication, 
awareness and involvement with persons and groups '-.'ithin and 
outside the immediate environment from its earliest moments. Th 
pro.iect staff made efforts to induce interest and cooperation from 
a wide group of supporters and potential users. Interaction and 
communication proceeded at a high rate through both formal and informal 
channels. Potential obstacles were shared concerns. The 
resolution processes often provided unanticipated benefit and 
strengthened the project. The development and maintenance of a 
network of communication took time and effort, but observable rewards 
justified the expenditure. The research was designed by the 
principal investigator, who dcvoted full time to the project. The 
host agency indicated its commitment by contributions of services 
and supplementary funds. The focus of the research was airaed at 
a felt need which enjoyed a shared interest fro^x oth'-rr people. 
Ipso facto, therefore, the product was readi' -k^table. Potential 
consumers v<^re involved and informed. They -louraged early efforts 
at dissemination of findings, and were ready \iO ccasider implications 
for utilization...^ 

6. The evaluator should know frun* the beginning of the project 
who will be responsible for planning vnd executing various change 
processes once the report is delivered. There should be an understanding 
as to 1) who shall copy the reporr in sufficient numbers for 
distribution; 2) who shall distribute the report; 3) who shall contribute 

6Glaset, E.M. and Taylor, S.H. "Factors influencing the success of 
applied research: a study of ten NIMH-funded projects." Human 
Interaction Research Institute, Los Angeles, 1969. 
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to the distribution list; 4) who is responsible for briefing 
the staff, the governing board, advisory groups, the public, the 
press, other agencies, etc.; 5) who is responsible for formulating 
recommendations and other future directions; 6) who ovms the raw 
data, the summary data, computer tapes, the report, and what is 
permissible for each, and their accessibility. 
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